From 6a559e5a341cf007889d46f59160c8914a84830e Mon Sep 17 00:00:00 2001 From: vj926 Date: Fri, 22 May 2026 09:04:21 -0700 Subject: [PATCH 1/4] deploy: add Azure Kubernetes Service tutorial for sidecar dev sample Adds an AKS variant of the dev (Ollama) sidecar deployment alongside the existing Azure Container Apps tutorial. Mirrors the upstream layout: - deploy/azure/kubernetes-service/dev/README.md - Microsoft Learn-style walkthrough (Phase 1 Entra -> Phase 7 Verify, cost, teardown, troubleshooting, kind smoke-test appendix). - .claude/skills/deploy-agent-aks-dev/ - automated fast-path skill (orchestrator, per-phase scripts, envsubst-rendered manifests, references covering SKU sizing, workload identity, cross-tenant federation, post-deploy manual steps, troubleshooting). - .claude/skills/teardown-agent-aks-dev/ - matching teardown skill. Key architectural points: - Secretless: agent pod uses Azure Workload Identity (federated identity credential on the Blueprint app) to acquire tokens; no client secret in the cluster. - Service account 'agent-sa' in namespace 'agentid' is the federation subject (system:serviceaccount:agentid:agent-sa). - Cross-tenant supported: Entra tenant (Blueprint + Agent apps) and Azure subscription tenant (AKS + ACR) may differ; FIC trust is OIDC-URL-based. - Default path is autonomous app-only auth; user-OBO is documented as an optional add-on via port-forward in section 11.4. Validated end-to-end on an AKS Standard_D4s_v5 cluster with Ollama llama3.2:3b. Tool-calling reliability table and SKU warnings document the silent-failure modes for under-sized models. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .claude/skills/deploy-agent-aks-dev/SKILL.md | 292 +++++++ .../manifests/00-namespace.yaml | 6 + .../manifests/10-serviceaccount.yaml | 16 + .../manifests/20-weather-api.yaml | 49 ++ .../manifests/30-ollama.yaml | 89 +++ .../manifests/40-agent.yaml | 81 ++ .../manifests/50-ingress.yaml | 14 + .../references/20-weather-api.yaml | 49 ++ .../references/40-agent.yaml | 81 ++ .../references/architecture.md | 57 ++ .../references/cross-tenant-federation.md | 64 ++ .../references/non-azure-k8s.md | 68 ++ .../references/post-deploy-manual-steps.md | 66 ++ .../references/sku-sizing.md | 82 ++ .../references/smoke-test.md | 65 ++ .../references/troubleshooting.md | 97 +++ .../references/workload-identity.md | 80 ++ .../scripts/01-create-aks.sh | 47 ++ .../scripts/02-build-and-push.sh | 42 + .../scripts/03-federate-blueprint.ps1 | 42 + .../scripts/04-apply-manifests.sh | 48 ++ .../scripts/add-spa-redirect-uri.sh | 83 ++ .../scripts/deploy-aks-dev.sh | 82 ++ .../scripts/deploy-vars.sh.template | 105 +++ .../scripts/grant-agent-obo-consent.ps1 | 42 + .../scripts/port-forward.sh | 41 + .../scripts/setup-obo-blueprint-for-aks.ps1 | 83 ++ .../scripts/smoke-test-kind.sh | 145 ++++ .../skills/teardown-agent-aks-dev/SKILL.md | 180 +++++ .../scripts/teardown-aks-dev.sh | 234 ++++++ deploy/azure/kubernetes-service/dev/README.md | 718 ++++++++++++++++++ 31 files changed, 3148 insertions(+) create mode 100644 .claude/skills/deploy-agent-aks-dev/SKILL.md create mode 100644 .claude/skills/deploy-agent-aks-dev/manifests/00-namespace.yaml create mode 100644 .claude/skills/deploy-agent-aks-dev/manifests/10-serviceaccount.yaml create mode 100644 .claude/skills/deploy-agent-aks-dev/manifests/20-weather-api.yaml create mode 100644 .claude/skills/deploy-agent-aks-dev/manifests/30-ollama.yaml create mode 100644 .claude/skills/deploy-agent-aks-dev/manifests/40-agent.yaml create mode 100644 .claude/skills/deploy-agent-aks-dev/manifests/50-ingress.yaml create mode 100644 .claude/skills/deploy-agent-aks-dev/references/20-weather-api.yaml create mode 100644 .claude/skills/deploy-agent-aks-dev/references/40-agent.yaml create mode 100644 .claude/skills/deploy-agent-aks-dev/references/architecture.md create mode 100644 .claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md create mode 100644 .claude/skills/deploy-agent-aks-dev/references/non-azure-k8s.md create mode 100644 .claude/skills/deploy-agent-aks-dev/references/post-deploy-manual-steps.md create mode 100644 .claude/skills/deploy-agent-aks-dev/references/sku-sizing.md create mode 100644 .claude/skills/deploy-agent-aks-dev/references/smoke-test.md create mode 100644 .claude/skills/deploy-agent-aks-dev/references/troubleshooting.md create mode 100644 .claude/skills/deploy-agent-aks-dev/references/workload-identity.md create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/01-create-aks.sh create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/03-federate-blueprint.ps1 create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/04-apply-manifests.sh create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/deploy-aks-dev.sh create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/deploy-vars.sh.template create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 create mode 100644 .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh create mode 100644 .claude/skills/teardown-agent-aks-dev/SKILL.md create mode 100644 .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh create mode 100644 deploy/azure/kubernetes-service/dev/README.md diff --git a/.claude/skills/deploy-agent-aks-dev/SKILL.md b/.claude/skills/deploy-agent-aks-dev/SKILL.md new file mode 100644 index 0000000..8282b48 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/SKILL.md @@ -0,0 +1,292 @@ +--- +name: deploy-agent-aks-dev +description: 'AI-led, end-to-end deployment of an agent that authenticates with Microsoft Entra Agent ID to Azure Kubernetes Service, using Azure Workload Identity instead of client secrets. Use when an engineering team wants to host their own agent (or this repo''s `sidecar/dev` sample) on AKS with the Entra Agent ID auth-sidecar pattern; when promoting an existing docker-compose stack from ClientSecret to secretless federation; or when an organization already standardized on Kubernetes and needs Agent ID to fit alongside their other workloads. Includes a kind-based local smoke test (no Azure cost), a one-shot orchestrator, a port-forward workflow for the OBO sign-in flow, and an explicit "Adapt for your own agent" section. NOT for Azure Container Apps (use `deploy-agent-aca-dev`), App Service (use `deploy-agent-appservice-dev`), or the AWS Bedrock variant (use `deploy-agent-aca-aws`). Chains to `entra-agent-id-setup` for the Blueprint + Agent Identity + Client SPA objects, and pairs with `teardown-agent-aks-dev` for cleanup.' +--- + +# Deploy an Entra Agent ID Agent to Azure Kubernetes Service (AI-Led) + +End-to-end, **secretless** deployment of an agent that uses the Microsoft Entra Agent ID auth-sidecar on AKS. The included `sidecar/dev` sample is the default runnable artifact; the skill also walks an external team through **adapting their own agent** to the same pattern. + +One federation chain — Kubernetes ServiceAccount → Blueprint app. **No client secrets in the cluster.** No UAMI in the middle. + +**Canonical manifests:** [`manifests/`](./manifests/). Long-form walkthrough tutorial: [`deploy/azure/kubernetes-service/dev/README.md`](../../../deploy/azure/kubernetes-service/dev/README.md). + +## When to Use + +- An engineering team wants to host their own agent on AKS using the Entra Agent ID sidecar pattern (autonomous app-identity flow + optional On-Behalf-Of user flow). +- The team already has AKS as a platform standard and needs Agent ID to fit alongside other workloads — no new compute service. +- The team wants the **secretless** posture: Workload Identity, projected SA tokens, FIC trust on the Blueprint app — no client secrets stored in the cluster. +- The user has a docker-compose dev stack from `sidecar/dev` and wants to promote it to AKS without rewriting the agent code. +- A non-Azure k8s cluster (EKS, GKE, on-prem) is the eventual target — this skill produces a reference layout that is 95% portable; see [references/non-azure-k8s.md](./references/non-azure-k8s.md). + +## Do NOT Use When + +- **Azure Container Apps** is the target — use [`deploy-agent-aca-dev`](../deploy-agent-aca-dev/SKILL.md). +- **Azure App Service** is the target — use `deploy-agent-appservice-dev`. +- **AWS Bedrock** is the LLM backend — use [`deploy-agent-aca-aws`](../deploy-agent-aca-aws/SKILL.md). +- **Local laptop docker-compose** is sufficient — use `sidecar/dev/` directly with `docker compose up`. No federation needed. +- The team only needs the agent for a short demo with no Kubernetes plans — ACA is cheaper and simpler. + +## Prerequisites (verify BEFORE running anything) + +1. **Entra role** on the signing-in user, one of: `Global Administrator`, `Agent ID Administrator`, `Agent ID Developer`. If unsure, run the [`entra-agent-id-setup`](../entra-agent-id-setup/SKILL.md) skill first — it surfaces the role requirement and creates the Blueprint/Agent/SPA objects. +2. **Azure RBAC**: `Owner` or `Contributor` on the subscription where AKS will live, plus `User Access Administrator` if the AKS attach-ACR step needs to grant `AcrPull`. +3. **Tooling**: + - `az` ≥ 2.60 with `aks-preview` extension (`az extension add --name aks-preview`) + - `kubectl` ≥ 1.28 + - `pwsh` 7.4+ with `Microsoft.Graph.Authentication` (`Install-Module Microsoft.Graph.Authentication -Scope CurrentUser`) + - `envsubst` (from the `gettext` package; on Windows comes with Git Bash) + - Optional for local smoke test: Docker Desktop + `kind` ≥ 0.20 + - Optional for "Adapt for your own agent": a container image of the user's agent in any registry reachable by AKS +4. **Tenant + subscription confirmed with the user.** ALWAYS confirm before any `az` command that mutates resources. Users frequently have multiple tenants; pick the wrong one and you create a half-deployed cluster in the wrong place. +5. **Entra Agent ID base objects exist** — Blueprint, Agent Identity, and (for OBO) a Client SPA. If not, chain `entra-agent-id-setup` first. +6. **Resource providers registered** on first use of a fresh subscription: + `Microsoft.ContainerService`, `Microsoft.ContainerRegistry`, `Microsoft.Compute`, `Microsoft.Network`, `Microsoft.Storage`, `Microsoft.OperationalInsights`, `Microsoft.OperationsManagement`. `01-create-aks.sh` checks and registers what's missing. + +> [!NOTE] +> **Windows / PowerShell users:** the orchestrator and scripts are bash + `pwsh`. Run them from **Git Bash** or **WSL**, not raw PowerShell — `source`, `envsubst`, and curl-style heredocs do not have native PowerShell equivalents. + +> [!NOTE] +> **Cross-tenant deployment** (the Azure subscription lives in tenant A while the Entra Agent ID objects live in tenant B) is supported. Set `SUBSCRIPTION_TENANT_ID` in `/tmp/deploy-vars.sh`. Full pattern: [references/cross-tenant-federation.md](./references/cross-tenant-federation.md). Default behavior is single-tenant. + +## SKU decisions — ask the user first + +Don't silently default to the cheapest tier. The orchestrator requires each SKU variable to be set and fails hard if any is missing. Confirm each value **with the user** before provisioning. Full tradeoff matrix and per-model sizing: [references/sku-sizing.md](./references/sku-sizing.md). + +| Variable | Ask | Demo default | Silent-failure mode if chosen wrong | +|---|---|---|---| +| `NODE_VM_SIZE` | demo / GPU / Spot | `Standard_D4s_v5` | `qwen2.5:7b` on `D2` → OOM kill; tokens come out at 1 char/sec on `B2s` | +| `NODE_COUNT` | 1 – 5 | `2` | `1` = no headroom during model pulls; if the node restarts, every pod becomes `Pending` | +| `ACR_SKU` | `Basic` / `Standard` / `Premium` | `Basic` | Basic's 10 GB fills up with ~6 baked-Ollama variants | +| `STORAGE_GB` | PVC size for Ollama models | `20` | 1B model = 1.3 GB; 7B = 4.3 GB; too small → init container hangs on disk-full | +| `OLLAMA_MODEL` | `qwen2.5:1.5b` / `qwen2.5:7b` / `llama3.2:1b` / your own | `qwen2.5:1.5b` | 7B on CPU node = 30 s+ per turn; tool-calling becomes unreliable | +| `INGRESS_TYPE` | `LoadBalancer` / `ingress-nginx` / `appgw` | `LoadBalancer` | `nginx` needs Helm + cert-manager; `appgw` adds ~$240/mo | +| `ENABLE_LOGS` | `none` / `azure-monitor-container-insights` | `none` | `none` hides crash loops; flip on once you hit a "why" moment | + +**When invoking this skill, explicitly state the defaults to the user and ask them to confirm or override — do not assume.** + +## Procedure + +The procedure is built for the included `sidecar/dev` sample. If you're bringing your own agent, do Steps 0–3 unchanged and then jump to the **[Adapt for your own agent](#adapt-for-your-own-agent)** section before Step 4. + +### Step 0 — Confirm account and populate variables + +```bash +cp .claude/skills/deploy-agent-aks-dev/scripts/deploy-vars.sh.template /tmp/deploy-vars.sh +# Edit /tmp/deploy-vars.sh: fill in TENANT_ID, SUBSCRIPTION_ID, RG, LOCATION, SKUs. +source /tmp/deploy-vars.sh + +az login --tenant "${SUBSCRIPTION_TENANT_ID:-$TENANT_ID}" +az account set --subscription "$SUBSCRIPTION_ID" +az account show --query '{name:name, id:id, tenantId:tenantId}' -o table +``` + +Stop and confirm with the user before proceeding. Wrong-tenant deployments are the #1 source of cleanup pain. + +### Step 1 — Create Entra Agent ID base objects + +Delegate to [`entra-agent-id-setup`](../entra-agent-id-setup/SKILL.md). Capture `BLUEPRINT_APP_ID`, `AGENT_CLIENT_ID`, and (for OBO) `CLIENT_SPA_APP_ID` into `/tmp/deploy-vars.sh`. + +Then configure the Blueprint for OBO (sets `identifierUris`, adds the `access_as_user` scope, pre-authorizes the Client SPA, and pre-grants admin consent — all idempotent): + +```bash +pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 \ + -BlueprintAppId "$BLUEPRINT_APP_ID" \ + -ClientSpaAppId "$CLIENT_SPA_APP_ID" \ + -AgentAppId "$AGENT_CLIENT_ID" \ + -TenantId "$TENANT_ID" +``` + +Skip this script if the deployment is autonomous-only (no user sign-in). It's safe to run twice; subsequent runs are no-ops. + +### Step A — Local smoke test on `kind` (RECOMMENDED before Azure) + +Validate every manifest against a real Kubernetes API server with no Azure cost. The smoke test uses `ClientSecret` for the sidecar (matches the upstream docker-compose), so no federation is required. + +```bash +bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh +``` + +What it covers and what it doesn't: [references/smoke-test.md](./references/smoke-test.md). Output is one line — `SMOKE PASS` or `SMOKE FAIL: `. **Do this before Step 2** unless you're already comfortable with the manifests. + +### Step 2 — Azure infrastructure (RG + ACR + AKS with OIDC + Workload Identity) + +```bash +bash .claude/skills/deploy-agent-aks-dev/scripts/01-create-aks.sh +``` + +Creates: +- Resource group in `$LOCATION`. +- ACR `$ACR_NAME` with `--admin-enabled false`. +- AKS `$AKS_NAME` with `--enable-oidc-issuer --enable-workload-identity` and a system pool of `$NODE_COUNT × $NODE_VM_SIZE`. +- `az aks update --attach-acr` → the kubelet's MI gets `AcrPull` on the ACR (no `imagePullSecrets` needed). +- Appends `OIDC_ISSUER=` to `/tmp/deploy-vars.sh`. + +> [!NOTE] +> If your tenant enforces Azure Policy that blocks public LBs (common in regulated environments), set `INGRESS_TYPE=ingress-nginx` and install the controller manually. The LoadBalancer Service in `50-ingress.yaml` becomes a ClusterIP + Ingress. + +### Step 3 — Federate the KSA to the Blueprint app (the only federation chain) + +```bash +pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/03-federate-blueprint.ps1 \ + -TenantId "$TENANT_ID" \ + -BlueprintAppId "$BLUEPRINT_APP_ID" \ + -OidcIssuerUrl "$OIDC_ISSUER" \ + -FicName "${FIC_NAME:-aks-agent-sa}" +``` + +Adds one Federated Identity Credential on the Blueprint app: +- `issuer` = the AKS OIDC URL captured in Step 2 +- `subject` = `system:serviceaccount:agentid:agent-sa` +- `audience` = `api://AzureADTokenExchange` + +**No UAMI is created.** The pod's projected SA token IS the assertion the Blueprint accepts. See [references/workload-identity.md](./references/workload-identity.md) for why this works and what `SignedAssertionFilePath` does. + +### Step 4 — Build and push container images + +```bash +bash .claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh +``` + +`az acr build` for `llm-agent` and `weather-api`. **No local Docker required.** Ollama uses the upstream `ollama/ollama:latest` image as-is — the model is fetched by an initContainer on first pod start and persisted in a PVC. If your tenant has an Azure Policy blocking public Docker Hub pulls, pre-import: `az acr import --name "$ACR_NAME" --source docker.io/ollama/ollama:latest` and update `30-ollama.yaml` to reference the ACR copy. + +### Step 5 — Apply manifests + +```bash +bash .claude/skills/deploy-agent-aks-dev/scripts/04-apply-manifests.sh +``` + +Renders `manifests/*.yaml` through `envsubst` (with an **explicit varlist** — `$TENANT_ID $BLUEPRINT_APP_ID $AGENT_CLIENT_ID $ACR_NAME $OLLAMA_MODEL $CLIENT_SPA_APP_ID` — to avoid clobbering shell variables like `$PID` inside init scripts), then `kubectl apply -f -`, then `kubectl rollout status` per Deployment, and finally waits for the LoadBalancer external IP. Captures `APP_FQDN=` into `/tmp/deploy-vars.sh`. + +### Step 6 — Post-deploy wiring (REQUIRED — execute immediately after Step 5) + +> [!IMPORTANT] +> AI agent **MUST** execute all three sub-steps. Do NOT skip. Full rationale + portal fallback: [references/post-deploy-manual-steps.md](./references/post-deploy-manual-steps.md). + +1. **Add SPA redirect URIs.** Registers `http://localhost:8080/` (required for OBO — see sub-step 3) and, if `APP_FQDN` is set, also `http://$APP_FQDN/`: + + ```bash + APP_FQDN="$APP_FQDN" \ + bash .claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh + ``` + +2. **Grant Agent → Graph delegated `User.Read`** (fixes `AADSTS65001` on OBO): + + ```bash + pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 \ + -AgentAppId "$AGENT_CLIENT_ID" -TenantId "$TENANT_ID" + ``` + +3. **Use port-forward for OBO sign-in.** The LoadBalancer is plain HTTP, which browsers refuse to treat as a "secure context" — MSAL's PKCE flow needs `crypto.subtle`, which is gated on secure-context, so the sign-in popup never opens on `http://`. Loopback is exempt: + + ```bash + bash .claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh + # browser: http://localhost:8080 → click "Sign In" + ``` + + Autonomous (no-sign-in) mode works fine on the raw `http://$APP_FQDN/` without port-forward. + +### Step 7 — Verify + +```bash +# Sidecar reachable on localhost from the agent container +kubectl -n agentid exec deploy/llm-agent -c llm-agent -- \ + curl -fsS http://localhost:5000/AuthorizationHeader?api=graph-app | head -c 80 + +# End-to-end autonomous flow +curl -fsS "http://$APP_FQDN/status" # expect: ollama_available: true, sidecar_reachable: true + +# Workload identity wired? +kubectl -n agentid exec deploy/llm-agent -c sidecar -- \ + ls /var/run/secrets/azure/tokens/ # expect: azure-identity-token +kubectl -n agentid exec deploy/llm-agent -c sidecar -- env | grep AZURE_ +# expect AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE, AZURE_AUTHORITY_HOST +``` + +Then open `http://$APP_FQDN/` in a browser. Autonomous mode should return a model-generated answer ("weather in Seattle?" → real Open-Meteo data via the weather-api). Then run the port-forward and sign in via the MSAL popup to exercise OBO. + +## Adapt for your own agent + +The pattern is: **your agent container + the auth-sidecar container, in one pod, sharing localhost. The KSA federates to your Blueprint app. The sidecar reads the projected SA token from disk and signs Entra assertions on every outbound call.** + +To swap in your own agent (instead of `llm-agent`): + +1. **Build your agent image** to a registry AKS can pull from (the ACR created in Step 2 works; so does any other registry as long as the kubelet has pull credentials). + +2. **Replace the `llm-agent` container spec** in `manifests/40-agent.yaml` with your image. Keep: + - The pod label `azure.workload.identity/use: "true"` (triggers the webhook to project the SA token). + - The KSA `agent-sa` (annotated with the Blueprint's `client-id` and the tenant). + - The `sidecar` container, byte-for-byte unchanged. Image: `mcr.microsoft.com/entra-sdk/auth-sidecar:1.0.0-azurelinux3.0-distroless`. It listens on `localhost:5000`. + +3. **Call the sidecar from your agent code** for every outbound token need: + ```http + GET http://localhost:5000/AuthorizationHeader?api= + → 200 OK + Authorization: Bearer eyJ... + ``` + The `` matches a `DownstreamApis____*` env var on the sidecar (see `40-agent.yaml`). To add a downstream API, set: + ```yaml + - name: AzureAd__ClientCredentials__0__SourceType + value: SignedAssertionFilePath # do not change + - name: DownstreamApis__myapi__BaseUrl + value: https://api.example.com + - name: DownstreamApis__myapi__Scopes__0 + value: api://your-api-app-id/.default + - name: DownstreamApis__myapi__RequestAppToken + value: "true" # app-only; remove for OBO + ``` + +4. **Re-issue federation if your namespace / KSA differ** from `agentid` / `agent-sa`. The FIC's `subject` field is exact-match — update Step 3 inputs. + +5. **(Optional) Replace `weather-api`** with your downstream API. The sidecar handles token validation on the caller side; the API must validate the JWT (issuer, audience = its own app ID URI, `appid` claim = the Agent ID app ID). See `manifests/20-weather-api.yaml` for a Python reference implementation. + +6. **(Optional) Replace `ollama`** with Azure OpenAI or any other completions backend by removing the `ollama` Deployment/Service/PVC and updating your agent's `OLLAMA_URL` (or analogous) env var. + +What you do NOT need to change: the FIC, the sidecar image, the KSA annotations, the pod label, the projected-token volume path. Those are the contract between AKS Workload Identity and Entra Agent ID — and the contract is what makes this pattern portable across agents. + +## One-Shot Orchestrator + +When prereqs are met and SKU variables confirmed: + +```bash +source /tmp/deploy-vars.sh +bash .claude/skills/deploy-agent-aks-dev/scripts/deploy-aks-dev.sh +``` + +Idempotent. Runs Steps 2 → 5 in order. Steps 0, 1, A, 6 require human decisions or interactive sign-in and remain manual. + +## Key Artifacts + +Persisted in `/tmp/deploy-vars.sh`: + +| Variable | Source | Required? | +|---|---|---| +| `TENANT_ID`, `SUBSCRIPTION_ID` | User | always | +| `SUBSCRIPTION_TENANT_ID` | User | only for cross-tenant deployments | +| `RG`, `LOCATION`, `AKS_NAME`, `ACR_NAME`, `NODE_COUNT`, `NODE_VM_SIZE` | User (SKU) | always | +| `OLLAMA_MODEL`, `STORAGE_GB`, `INGRESS_TYPE`, `ENABLE_LOGS` | User (SKU) | always | +| `BLUEPRINT_APP_ID`, `AGENT_CLIENT_ID` | Step 1 (`entra-agent-id-setup`) | always | +| `CLIENT_SPA_APP_ID` | Step 1 | only for OBO | +| `BLUEPRINT_CLIENT_SECRET` | Step 1 | only for `kind` smoke test | +| `OIDC_ISSUER` | Step 2 | autofilled | +| `APP_FQDN` (= LoadBalancer IP) | Step 5 | autofilled | +| `FIC_NAME` | User (optional) | defaults to `aks-agent-sa`; set when redeploying to avoid collision with old FICs | + +**No client secrets in the cluster.** The only secret on disk is the projected SA token, rotated automatically by the kubelet ~10 minutes before expiry. + +## References + +- [Architecture summary](./references/architecture.md) — pod / sidecar / Service / federation diagram +- [SKU and sizing decisions](./references/sku-sizing.md) — node, ACR, Ollama, ingress, logs cost matrix +- [Workload Identity deep-dive](./references/workload-identity.md) — why `SignedAssertionFilePath` works and how the FIC is validated +- [Cross-tenant federation](./references/cross-tenant-federation.md) — Azure sub in tenant A, Entra objects in tenant B +- [Post-deploy manual steps](./references/post-deploy-manual-steps.md) — SPA redirect URI, OBO consent, port-forward rationale +- [Adapting to EKS / GKE / on-prem](./references/non-azure-k8s.md) — the only Azure-specific pieces and what replaces them +- [Local smoke test on `kind`](./references/smoke-test.md) — what's covered, what's not, how to interpret failures +- [Troubleshooting matrix](./references/troubleshooting.md) — symptom → cause → fix tables + +## Paired skills + +- **Setup of Entra objects:** [`entra-agent-id-setup`](../entra-agent-id-setup/SKILL.md) — creates Blueprint + Agent Identity + Client SPA. +- **Teardown:** [`teardown-agent-aks-dev`](../teardown-agent-aks-dev/SKILL.md) — reverses this skill. DRY-RUN by default. Cleans the RG, the FIC on the Blueprint, and (opt-in) the Entra apps. +- **Alternate hosting:** [`deploy-agent-aca-dev`](../deploy-agent-aca-dev/SKILL.md) — same agent, Azure Container Apps instead of AKS. Use when the team is not already on Kubernetes. diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/00-namespace.yaml b/.claude/skills/deploy-agent-aks-dev/manifests/00-namespace.yaml new file mode 100644 index 0000000..f699be0 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/manifests/00-namespace.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: agentid + labels: + app.kubernetes.io/part-of: entra-agentid-sidecar-dev diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/10-serviceaccount.yaml b/.claude/skills/deploy-agent-aks-dev/manifests/10-serviceaccount.yaml new file mode 100644 index 0000000..f29c877 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/manifests/10-serviceaccount.yaml @@ -0,0 +1,16 @@ +# Workload-identity-enabled ServiceAccount. +# Federation is created by scripts/03-federate-blueprint.ps1: +# issuer = AKS cluster OIDC issuer +# subject = system:serviceaccount:agentid:agent-sa +# audience = api://AzureADTokenExchange +# The Blueprint app trusts this subject directly — no UAMI in the middle. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: agent-sa + namespace: agentid + annotations: + # client-id of the BLUEPRINT app (NOT the agent app). + # Replaced by 04-apply-manifests.sh via envsubst. + azure.workload.identity/client-id: "${BLUEPRINT_APP_ID}" + azure.workload.identity/tenant-id: "${TENANT_ID}" diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/20-weather-api.yaml b/.claude/skills/deploy-agent-aks-dev/manifests/20-weather-api.yaml new file mode 100644 index 0000000..ab80624 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/manifests/20-weather-api.yaml @@ -0,0 +1,49 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: weather-api + namespace: agentid +spec: + replicas: 1 + selector: + matchLabels: { app: weather-api } + template: + metadata: + labels: { app: weather-api } + spec: + containers: + - name: weather-api + image: "${ACR_NAME}.azurecr.io/agent-id-dev/weather-api:1.0.0" + ports: + - containerPort: 8080 + env: + - { name: TENANT_ID, value: "${TENANT_ID}" } + - { name: VALIDATE_TOKEN_SIGNATURE, value: "true" } + volumeMounts: + - name: patched-app + mountPath: /app/app.py + subPath: app.py + resources: + requests: { cpu: "100m", memory: "128Mi" } + limits: { cpu: "500m", memory: "512Mi" } + readinessProbe: + httpGet: { path: /health, port: 8080 } + initialDelaySeconds: 5 + periodSeconds: 10 + volumes: + - name: patched-app + configMap: + name: weather-api-patch +--- +apiVersion: v1 +kind: Service +metadata: + name: weather-api + namespace: agentid +spec: + type: ClusterIP + selector: { app: weather-api } + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/30-ollama.yaml b/.claude/skills/deploy-agent-aks-dev/manifests/30-ollama.yaml new file mode 100644 index 0000000..ee4b578 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/manifests/30-ollama.yaml @@ -0,0 +1,89 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ollama-models + namespace: agentid +spec: + accessModes: [ReadWriteOnce] + storageClassName: managed-csi + resources: + requests: + storage: 20Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ollama + namespace: agentid +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: { app: ollama } + template: + metadata: + labels: { app: ollama } + spec: + # Init container pulls the model once into the PVC so the main + # container starts fast and the model survives pod restarts. + initContainers: + - name: model-puller + image: ollama/ollama:latest + command: ["/bin/sh","-c"] + # Use $OLLAMA_MODEL (no braces) so envsubst leaves runtime expansion + # to bash. With ${OLLAMA_MODEL}, envsubst would bake the model name + # into the args at apply time, defeating future `kubectl set env`. + args: + - | + set -e + ollama serve & + PID=$! + until ollama list >/dev/null 2>&1; do sleep 1; done + if ! ollama list | awk '{print $1}' | grep -q "^$OLLAMA_MODEL$"; then + echo "Pulling $OLLAMA_MODEL..." + ollama pull "$OLLAMA_MODEL" + else + echo "$OLLAMA_MODEL already present, skipping pull." + fi + kill $PID + wait $PID 2>/dev/null || true + env: + - { name: OLLAMA_MODEL, value: "${OLLAMA_MODEL}" } + - { name: OLLAMA_HOST, value: "0.0.0.0:11434" } + - { name: HOME, value: "/root" } + volumeMounts: + - { name: models, mountPath: /root/.ollama } + containers: + - name: ollama + image: ollama/ollama:latest + ports: + - containerPort: 11434 + env: + - { name: OLLAMA_HOST, value: "0.0.0.0:11434" } + volumeMounts: + - { name: models, mountPath: /root/.ollama } + resources: + requests: { cpu: "500m", memory: "2Gi" } + limits: { cpu: "2", memory: "4Gi" } + readinessProbe: + tcpSocket: { port: 11434 } + initialDelaySeconds: 10 + periodSeconds: 10 + volumes: + - name: models + persistentVolumeClaim: + claimName: ollama-models +--- +apiVersion: v1 +kind: Service +metadata: + name: ollama + namespace: agentid +spec: + type: ClusterIP + selector: { app: ollama } + ports: + - name: http + port: 11434 + targetPort: 11434 diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/40-agent.yaml b/.claude/skills/deploy-agent-aks-dev/manifests/40-agent.yaml new file mode 100644 index 0000000..3f526ce --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/manifests/40-agent.yaml @@ -0,0 +1,81 @@ +# Agent pod = llm-agent + auth-sidecar (true sidecar, same trust boundary). +# weather-api and ollama are reached via in-cluster Services. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: llm-agent + namespace: agentid +spec: + replicas: 1 + selector: + matchLabels: { app: llm-agent } + template: + metadata: + labels: + app: llm-agent + # Required for the Azure Workload Identity mutating webhook to + # inject AZURE_* env vars and the projected SA token volume. + azure.workload.identity/use: "true" + spec: + serviceAccountName: agent-sa + containers: + - name: llm-agent + image: "${ACR_NAME}.azurecr.io/agent-id-dev/llm-agent:1.0.0" + ports: + - containerPort: 3000 + env: + - { name: TENANT_ID, value: "${TENANT_ID}" } + - { name: BLUEPRINT_APP_ID, value: "${BLUEPRINT_APP_ID}" } + - { name: AGENT_APP_ID, value: "${AGENT_CLIENT_ID}" } + - { name: AGENT_CLIENT_ID, value: "${AGENT_CLIENT_ID}" } + - { name: CLIENT_SPA_APP_ID, value: "${CLIENT_SPA_APP_ID}" } + - { name: SIDECAR_URL, value: "http://localhost:5000" } + - { name: WEATHER_API_URL, value: "http://weather-api.agentid.svc.cluster.local:8080" } + - { name: OLLAMA_URL, value: "http://ollama.agentid.svc.cluster.local:11434" } + - { name: OLLAMA_MODEL, value: "${OLLAMA_MODEL}" } + resources: + requests: { cpu: "200m", memory: "256Mi" } + limits: { cpu: "1", memory: "1Gi" } + volumeMounts: + - name: patched-app + mountPath: /app/app.py + subPath: app.py + readinessProbe: + httpGet: { path: /, port: 3000 } + initialDelaySeconds: 10 + periodSeconds: 10 + + - name: sidecar + image: mcr.microsoft.com/entra-sdk/auth-sidecar:1.0.0-azurelinux3.0-distroless + # Sidecar listens on localhost only — never exposed via Service. + env: + - { name: AzureAd__Instance, value: "https://login.microsoftonline.com/" } + - { name: AzureAd__TenantId, value: "${TENANT_ID}" } + - { name: AzureAd__ClientId, value: "${BLUEPRINT_APP_ID}" } + # Workload Identity: the projected SA token IS the signed + # assertion the Blueprint app's FIC accepts. Read it from disk. + - { name: AzureAd__ClientCredentials__0__SourceType, + value: "SignedAssertionFilePath" } + - { name: AzureAd__ClientCredentials__0__SignedAssertionFileDiskPath, + value: "/var/run/secrets/azure/tokens/azure-identity-token" } + # Autonomous (app-only) downstream + - { name: DownstreamApis__graph-app__BaseUrl, + value: "https://graph.microsoft.com/v1.0/" } + - { name: DownstreamApis__graph-app__Scopes__0, + value: "https://graph.microsoft.com/.default" } + - { name: DownstreamApis__graph-app__RequestAppToken, + value: "true" } + # OBO downstream + - { name: DownstreamApis__graph__BaseUrl, + value: "https://graph.microsoft.com/v1.0/" } + - { name: DownstreamApis__graph__Scopes__0, + value: "https://graph.microsoft.com/.default" } + - { name: ASPNETCORE_ENVIRONMENT, value: "Production" } + - { name: ASPNETCORE_URLS, value: "http://+:5000" } + resources: + requests: { cpu: "100m", memory: "128Mi" } + limits: { cpu: "500m", memory: "512Mi" } + volumes: + - name: patched-app + configMap: + name: llm-agent-patch diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/50-ingress.yaml b/.claude/skills/deploy-agent-aks-dev/manifests/50-ingress.yaml new file mode 100644 index 0000000..a4e9617 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/manifests/50-ingress.yaml @@ -0,0 +1,14 @@ +# LoadBalancer keeps things simple — no ingress controller required. +# Swap to type ClusterIP + an Ingress object if you have NGINX/AGIC installed. +apiVersion: v1 +kind: Service +metadata: + name: llm-agent + namespace: agentid +spec: + type: LoadBalancer + selector: { app: llm-agent } + ports: + - name: http + port: 80 + targetPort: 3000 diff --git a/.claude/skills/deploy-agent-aks-dev/references/20-weather-api.yaml b/.claude/skills/deploy-agent-aks-dev/references/20-weather-api.yaml new file mode 100644 index 0000000..ab80624 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/20-weather-api.yaml @@ -0,0 +1,49 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: weather-api + namespace: agentid +spec: + replicas: 1 + selector: + matchLabels: { app: weather-api } + template: + metadata: + labels: { app: weather-api } + spec: + containers: + - name: weather-api + image: "${ACR_NAME}.azurecr.io/agent-id-dev/weather-api:1.0.0" + ports: + - containerPort: 8080 + env: + - { name: TENANT_ID, value: "${TENANT_ID}" } + - { name: VALIDATE_TOKEN_SIGNATURE, value: "true" } + volumeMounts: + - name: patched-app + mountPath: /app/app.py + subPath: app.py + resources: + requests: { cpu: "100m", memory: "128Mi" } + limits: { cpu: "500m", memory: "512Mi" } + readinessProbe: + httpGet: { path: /health, port: 8080 } + initialDelaySeconds: 5 + periodSeconds: 10 + volumes: + - name: patched-app + configMap: + name: weather-api-patch +--- +apiVersion: v1 +kind: Service +metadata: + name: weather-api + namespace: agentid +spec: + type: ClusterIP + selector: { app: weather-api } + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/.claude/skills/deploy-agent-aks-dev/references/40-agent.yaml b/.claude/skills/deploy-agent-aks-dev/references/40-agent.yaml new file mode 100644 index 0000000..3f526ce --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/40-agent.yaml @@ -0,0 +1,81 @@ +# Agent pod = llm-agent + auth-sidecar (true sidecar, same trust boundary). +# weather-api and ollama are reached via in-cluster Services. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: llm-agent + namespace: agentid +spec: + replicas: 1 + selector: + matchLabels: { app: llm-agent } + template: + metadata: + labels: + app: llm-agent + # Required for the Azure Workload Identity mutating webhook to + # inject AZURE_* env vars and the projected SA token volume. + azure.workload.identity/use: "true" + spec: + serviceAccountName: agent-sa + containers: + - name: llm-agent + image: "${ACR_NAME}.azurecr.io/agent-id-dev/llm-agent:1.0.0" + ports: + - containerPort: 3000 + env: + - { name: TENANT_ID, value: "${TENANT_ID}" } + - { name: BLUEPRINT_APP_ID, value: "${BLUEPRINT_APP_ID}" } + - { name: AGENT_APP_ID, value: "${AGENT_CLIENT_ID}" } + - { name: AGENT_CLIENT_ID, value: "${AGENT_CLIENT_ID}" } + - { name: CLIENT_SPA_APP_ID, value: "${CLIENT_SPA_APP_ID}" } + - { name: SIDECAR_URL, value: "http://localhost:5000" } + - { name: WEATHER_API_URL, value: "http://weather-api.agentid.svc.cluster.local:8080" } + - { name: OLLAMA_URL, value: "http://ollama.agentid.svc.cluster.local:11434" } + - { name: OLLAMA_MODEL, value: "${OLLAMA_MODEL}" } + resources: + requests: { cpu: "200m", memory: "256Mi" } + limits: { cpu: "1", memory: "1Gi" } + volumeMounts: + - name: patched-app + mountPath: /app/app.py + subPath: app.py + readinessProbe: + httpGet: { path: /, port: 3000 } + initialDelaySeconds: 10 + periodSeconds: 10 + + - name: sidecar + image: mcr.microsoft.com/entra-sdk/auth-sidecar:1.0.0-azurelinux3.0-distroless + # Sidecar listens on localhost only — never exposed via Service. + env: + - { name: AzureAd__Instance, value: "https://login.microsoftonline.com/" } + - { name: AzureAd__TenantId, value: "${TENANT_ID}" } + - { name: AzureAd__ClientId, value: "${BLUEPRINT_APP_ID}" } + # Workload Identity: the projected SA token IS the signed + # assertion the Blueprint app's FIC accepts. Read it from disk. + - { name: AzureAd__ClientCredentials__0__SourceType, + value: "SignedAssertionFilePath" } + - { name: AzureAd__ClientCredentials__0__SignedAssertionFileDiskPath, + value: "/var/run/secrets/azure/tokens/azure-identity-token" } + # Autonomous (app-only) downstream + - { name: DownstreamApis__graph-app__BaseUrl, + value: "https://graph.microsoft.com/v1.0/" } + - { name: DownstreamApis__graph-app__Scopes__0, + value: "https://graph.microsoft.com/.default" } + - { name: DownstreamApis__graph-app__RequestAppToken, + value: "true" } + # OBO downstream + - { name: DownstreamApis__graph__BaseUrl, + value: "https://graph.microsoft.com/v1.0/" } + - { name: DownstreamApis__graph__Scopes__0, + value: "https://graph.microsoft.com/.default" } + - { name: ASPNETCORE_ENVIRONMENT, value: "Production" } + - { name: ASPNETCORE_URLS, value: "http://+:5000" } + resources: + requests: { cpu: "100m", memory: "128Mi" } + limits: { cpu: "500m", memory: "512Mi" } + volumes: + - name: patched-app + configMap: + name: llm-agent-patch diff --git a/.claude/skills/deploy-agent-aks-dev/references/architecture.md b/.claude/skills/deploy-agent-aks-dev/references/architecture.md new file mode 100644 index 0000000..733ac41 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/architecture.md @@ -0,0 +1,57 @@ +# Architecture summary + +One pod runs the **agent + sidecar** on shared `localhost`. Everything else is a separate Service. + +``` + ┌──────────────────────── AKS cluster ────────────────────────┐ + user ──https──▶ │ Service/LB ─▶ Pod: llm-agent │ + │ ├─ llm-agent (Flask, port 3000) │ + │ └─ auth-sidecar (port 5000, localhost only) │ + │ │ │ + │ ▼ │ + │ reads /var/run/secrets/azure/tokens/azure-identity-token │ + │ via SignedAssertionFilePath credential source │ + │ │ + │ Service: weather-api ─▶ Pod: weather-api (8080) │ + │ Service: ollama ─▶ Pod: ollama (11434) + PVC │ + └──────────────────────────────────────────────────────────────┘ + │ + ▼ + KSA agentid/agent-sa ──FIC (audience api://AzureADTokenExchange)──▶ Blueprint app + │ + ▼ + Graph / weather-api +``` + +| Container | Role | Port | +|---|---|---| +| `llm-agent` | Flask + LangChain; calls Ollama Service for completions | 3000 | +| `auth-sidecar` | `mcr.microsoft.com/entra-sdk/auth-sidecar`; reads SA token, signs Entra assertions | 5000 (localhost) | +| `weather-api` | Validates Agent Identity JWT (JWKS, iss, aud, appid) on every request | 8080 | +| `ollama` | Local LLM server, model on PVC | 11434 | + +**One federation chain, one direction:** +``` +KSA → Blueprint → Graph / weather-api +``` +- `KSA → Blueprint` audience: `api://AzureADTokenExchange` (workload-identity standard) +- `Blueprint → downstream` audience: `https://graph.microsoft.com` or the weather-api app ID URI + +**What's NOT in this deployment** (compared to the AWS variant): +- No UAMI / intermediary Entra app +- No external cloud OIDC IdP +- No token refresher container +- No `AWS_*` / `BEDROCK_*` env vars +- No shared `EmptyDir` for JWT passing — the workload identity webhook handles projection + +**What rotates:** Agent Identity tokens (minutes), projected SA tokens (~1 h). **What's permanent:** the FIC on the Blueprint app. **What's local:** Ollama weights on a PVC. + +## Why the agent + sidecar are in the same pod + +Microsoft security guidance for the auth-sidecar: it MUST be reachable only inside the same trust boundary. A k8s pod shares a network namespace, so `localhost:5000` is reachable from the agent container but not from any other pod, node process, or off-cluster client. This is the k8s equivalent of "no host port" in compose / "shared revision" in ACA. + +## Why weather-api and ollama are separate Deployments + +Unlike ACA (one container app = one process group), k8s lets each workload scale and store independently: +- **weather-api** demonstrates real cross-pod token validation. The agent calls a different Service IP, the request crosses the pod boundary, and the API independently validates the JWT. +- **ollama** has a PVC and may grow to a GPU node pool. Coupling it to the agent pod would force a model reload on every agent restart and make GPU scheduling awkward. diff --git a/.claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md b/.claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md new file mode 100644 index 0000000..04e4c6d --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md @@ -0,0 +1,64 @@ +# Cross-tenant deployment + +The default assumption in this skill is that the Azure subscription (where AKS, ACR, the Resource Group live) and the Entra tenant (where the Blueprint, Agent Identity, Client SPA live) are the **same tenant**. But that's not required — this layout supports a common real-world case where: + +| Concern | Tenant | +|---|---| +| Entra Agent ID objects (Blueprint, Agent, SPA) | A "demo" or "ISV" tenant (e.g. `M365-*.onmicrosoft.com`) — where you have Agent ID Admin | +| Azure subscription that pays for AKS | A different "corp" tenant — where you have Owner/Contributor on a sub | + +This page is the deep-dive. Quick-start is the callout in `SKILL.md` Step 0. + +## Why it works + +**Workload Identity federation trusts an OIDC issuer URL, not a tenant.** The Federated Identity Credential (FIC) on the Blueprint app is: + +``` +issuer = https://eastus2.oic.prod-aks.azure.com/// +subject = system:serviceaccount:agentid:agent-sa +audiences = [ api://AzureADTokenExchange ] +``` + +Entra (in the demo tenant) checks: "is this JWT signed by a key listed in the issuer's JWKS, and do `iss`/`sub`/`aud` match a registered FIC on this app?" It does **not** care which tenant the issuer happens to live in. As long as the issuer URL is publicly reachable (AKS OIDC issuers are), the assertion is valid. + +The sidecar's `AZURE_TENANT_ID` is set to the **demo tenant** (where the Blueprint lives) — that's the STS that mints the Blueprint token, not the issuer's tenant. + +## Variable contract + +`deploy-vars.sh` has two tenant variables: + +| Variable | Tenant | Used by | +|---|---|---| +| `SUBSCRIPTION_TENANT_ID` | Corp tenant (AKS/ACR/RG) | `az login --tenant`, `az account set`, RG/AKS/ACR ARM calls | +| `TENANT_ID` | Demo tenant (Blueprint, Agent, SPA) | `Connect-MgGraph -TenantId`, FIC create on Blueprint, sidecar `AZURE_TENANT_ID`, manifests' `TENANT_ID` env var | + +If `SUBSCRIPTION_TENANT_ID` is unset, scripts assume single-tenant and use `TENANT_ID` for both. **Existing single-tenant deployments don't need to change anything.** + +## The two `az login` flows + +```bash +# 1. Sub-tenant context for all `az` resource calls +az login --tenant "$SUBSCRIPTION_TENANT_ID" +az account set --subscription "$SUBSCRIPTION_ID" + +# 2. Graph context for FIC create / SPA redirect URI patch (interactive, separate browser sign-in) +pwsh -NoProfile -Command "Connect-MgGraph -TenantId '$TENANT_ID' -Scopes 'Application.ReadWrite.All' -NoWelcome" +``` + +The Graph cache from step (2) persists on disk in `~/.mg/`. Subsequent `pwsh` processes reuse it silently as long as the token hasn't expired. + +For `add-spa-redirect-uri.sh` the script calls `az account get-access-token --tenant "$TENANT_ID" --resource graph` — this triggers a one-time interactive sign-in to the demo tenant the first time, then caches. + +## Common gotchas + +| Symptom | Cause | Fix | +|---|---|---| +| `az account set --subscription` fails with `Subscription not found` | Active `az` context is in the wrong tenant | `az login --tenant "$SUBSCRIPTION_TENANT_ID"` | +| `AADSTS50020: User account ... does not exist in tenant` when calling Graph | Token requested without `--tenant` | Pass `--tenant "$TENANT_ID"` to `az account get-access-token` or `Connect-MgGraph -TenantId $TENANT_ID` | +| FIC create fails with `Authorization_RequestDenied` | Connected to Graph in the wrong tenant | `Disconnect-MgGraph; Connect-MgGraph -TenantId $TENANT_ID` | +| Sidecar logs `AADSTS700016: Application not found in directory` | `AzureAd__TenantId` points at the corp tenant, but the Blueprint is in the demo tenant | Set the manifest's `TENANT_ID` env to `$TENANT_ID` (demo); leave `SUBSCRIPTION_TENANT_ID` only for ARM | +| RG / AKS / ACR fail to create with `SubscriptionNotFound` even though `az account show` is correct | First time using this sub — resource providers not registered | `az provider register -n Microsoft.ContainerService; ContainerRegistry; Compute; Network; Storage; OperationalInsights; OperationsManagement` | + +## Teardown caveat + +The companion `teardown-agent-aks-dev` skill uses the same split: `SUBSCRIPTION_TENANT_ID` for the RG delete, `TENANT_ID` for FIC delete on the Blueprint and (opt-in) Entra-object deletes. Keep both vars in `/tmp/deploy-vars.sh` so teardown can target them correctly. diff --git a/.claude/skills/deploy-agent-aks-dev/references/non-azure-k8s.md b/.claude/skills/deploy-agent-aks-dev/references/non-azure-k8s.md new file mode 100644 index 0000000..f7b8dc9 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/non-azure-k8s.md @@ -0,0 +1,68 @@ +# Adapting to non-Azure Kubernetes + +The manifests in [`../manifests/`](../manifests/) are 95% portable. Only the credential injection differs by platform. + +## What stays the same on EKS / GKE / on-prem + +- `00-namespace.yaml` +- `20-weather-api.yaml` +- `30-ollama.yaml` +- `40-agent.yaml` (the two containers and their env, except the sidecar's credential source) +- `50-ingress.yaml` (swap Service type for your platform's idiom) +- The federation Graph call in `03-federate-blueprint.ps1` — only `OidcIssuerUrl` and `subject` change. + +## What you change per platform + +### Amazon EKS (IRSA) + +EKS Pod Identity / IRSA exposes the SA token at `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`. Different path, same shape (signed by the cluster's OIDC issuer). + +1. Get your EKS OIDC issuer URL: `aws eks describe-cluster --name --query 'cluster.identity.oidc.issuer' --output text`. +2. Federate the Blueprint app with `issuer=`, `subject=system:serviceaccount:agentid:agent-sa`. +3. Override the sidecar credential source: + ```yaml + - name: AzureAd__ClientCredentials__0__SourceType + value: SignedAssertionFilePath + - name: AzureAd__ClientCredentials__0__SignedAssertionFileDiskPath + value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token + ``` +4. Drop the `azure.workload.identity/*` annotations / labels — EKS doesn't use them. +5. If you want IAM Roles for Service Accounts to also work for AWS-side calls, add the `eks.amazonaws.com/role-arn` annotation on the KSA. Not needed for the Entra side. + +### Google GKE (Workload Identity) + +GKE projects a token at `/var/run/service-account/token` (or the standard SA path, depending on Workload Identity version). + +1. Get the issuer: `gcloud container clusters describe --format='value(workloadIdentityConfig.workloadPool)'` — this gives the pool; the issuer URL is `https://container.googleapis.com/v1/projects//locations//clusters/`. +2. Federate the Blueprint with `issuer=`, `subject=system:serviceaccount:agentid:agent-sa`. +3. Same sidecar override as EKS, pointing at GKE's token path. + +### On-prem with self-managed OIDC + +Standard k8s ≥ 1.21 with `--service-account-issuer` and `--service-account-jwks-uri` flags configured. You must: +1. Expose `/.well-known/openid-configuration` and the JWKS publicly (Entra needs to fetch keys). +2. Make sure the issuer in the JWT matches what Entra will see. +3. Federate as above. + +## Why this works at all + +Entra Agent ID federation is **OIDC-standard, not Azure-specific**. Any token that: +1. Is signed by a key in a JWKS Entra can fetch. +2. Has `iss` matching what you put in the FIC. +3. Has `sub` matching what you put in the FIC. +4. Has `aud=api://AzureADTokenExchange`. + +…will be accepted. Workload identity on AKS, IRSA on EKS, Workload Identity on GKE, self-managed on-prem — they all produce conformant tokens. + +## What customers can copy as-is + +- The auth-sidecar container spec (just change the credential source values). +- `weather-api` and `ollama` Deployments + Services. +- The agent container spec. +- The federation script (only 3 inputs change). + +What they MUST author per platform: +- Cluster provisioning (already platform-specific). +- The pod-level annotation / label that triggers projection (Azure: webhook label; AWS: SA annotation; GCP: KSA annotation). +- Ingress. +- Image registry pull config (`imagePullSecrets` or platform equivalent). diff --git a/.claude/skills/deploy-agent-aks-dev/references/post-deploy-manual-steps.md b/.claude/skills/deploy-agent-aks-dev/references/post-deploy-manual-steps.md new file mode 100644 index 0000000..8d6e4aa --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/post-deploy-manual-steps.md @@ -0,0 +1,66 @@ +# Post-deploy manual steps + +Three steps cannot be completed before the cluster + LoadBalancer exist, and none can be done via `az ad app update`. Run them after Step 5 (apply manifests) in the main SKILL procedure. + +## 1. Add SPA redirect URIs + +The Client SPA app was registered with only `http://localhost:3003`. For browser sign-in to work, you need at least: + +| Redirect URI | Why | +|---|---| +| `http://localhost:8080/` | **Required for OBO sign-in.** Browsers refuse to run MSAL's PKCE on raw-IP HTTP because it's not a secure context; loopback is exempt. Used together with `kubectl port-forward`. | +| `http:///` | Optional. Lets the autonomous-mode UI load directly on the LoadBalancer IP. Sign-in will still fail from this origin — that's normal. | + +`az ad app update --web-redirect-uris` does NOT modify SPA URIs — you must PATCH Graph directly: + +```bash +APP_FQDN="$APP_FQDN" \ + bash .claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh +``` + +The script idempotently appends both `http://localhost:8080/` and `http://$APP_FQDN/` to `spa.redirectUris`. + +Portal fallback: **Microsoft Entra ID** → **App registrations** → *Client SPA* → **Authentication** → **Single-page application** → **Add URI** → enter `http://localhost:8080/` → **Save**. + +## 2. Grant Agent → Graph delegated `User.Read` admin consent + +### Symptom + +Browser OBO flow fails with: + +``` +AADSTS65001: The user or administrator has not consented to use the application +``` + +### Why + +`Start-EntraAgentIDWorkflow` grants **application** Graph permissions (e.g., `User.Read.All`) only. OBO additionally requires a **delegated** permission (`User.Read`) with admin consent at the tenant level, because the exchange happens on behalf of a user. + +### Fix + +```powershell +pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 ` + -AgentAppId "$env:AGENT_CLIENT_ID" -TenantId "$env:TENANT_ID" +``` + +Idempotent — checks for an existing grant first. Creates `oauth2PermissionGrant`: `clientId=`, `resourceId=`, `consentType=AllPrincipals`, `scope=User.Read`. + +## 3. Open the agent via port-forward to exercise OBO + +```bash +bash .claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh +# in another shell / browser: +# http://localhost:8080 +``` + +Then click **Sign In** — the MSAL popup should open and you can sign in as any user in `$TENANT_ID`. + +For autonomous mode (no sign-in needed), `http:///` works directly with no port-forward. + +## Why these aren't automated inside Step 5 + +- **SPA URI** depends on the LoadBalancer external IP, which exists only after `kubectl apply` + LB provisioning. Cannot be precomputed. +- **OBO consent** is intentionally separate because autonomous-only deployments don't need it. Bundling it would hide a tenant-level admin consent behind a generic deploy command. +- **Port-forward** is an interactive convenience, not a deploy step. Producing it as a long-running background process inside a deploy script would surprise the user. + +Scripts (1) and (2) are byte-identical (modulo filenames/paths) to the ACA skill's `add-spa-redirect-uri.sh` and `grant-agent-obo-consent.ps1` — they're Entra-level operations, not cloud-specific. diff --git a/.claude/skills/deploy-agent-aks-dev/references/sku-sizing.md b/.claude/skills/deploy-agent-aks-dev/references/sku-sizing.md new file mode 100644 index 0000000..4b2d636 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/sku-sizing.md @@ -0,0 +1,82 @@ +# SKU and sizing decisions + +Demo-cost target: **~$150/mo if left running 24×7**. Tear down between sessions. + +## Node pool + +| `NODE_VM_SIZE` | vCPU / RAM | ~$/mo (East US, PAYG) | Use case | +|---|---|---|---| +| `Standard_B2s` | 2 / 4 | ~$30 | Smallest possible demo; `qwen2.5:1.5b` works but is slow | +| `Standard_D2s_v5` | 2 / 8 | ~$70 | **Default**. Comfortable for `qwen2.5:1.5b`. | +| `Standard_D4s_v5` | 4 / 16 | ~$140 | Required for `qwen2.5:7b` on CPU | +| `Standard_NC4as_T4_v3` | 4 / 28 + T4 GPU | ~$540 | Required for `qwen2.5:7b` at usable latency | + +`NODE_COUNT=2` ensures one node free for rescheduling during model pulls or upgrades. Customer demos can drop to 1. + +## Ollama + +| `OLLAMA_MODEL` | RAM (peak) | Disk | First-token latency on D2s_v5 | +|---|---|---|---| +| `qwen2.5:0.5b` | ~700 MB | ~400 MB | <1 s | +| **`qwen2.5:1.5b`** | ~1.5 GB | ~1.3 GB | 1–2 s | +| `llama3.2:1b` | ~1.5 GB | ~1.3 GB | 1–2 s | +| `qwen2.5:3b` | ~2.5 GB | ~2 GB | 3–5 s | +| `qwen2.5:7b` (CPU) | ~5.5 GB | ~4.3 GB | 15–30 s (avoid) | +| `qwen2.5:7b` (GPU) | ~5.5 GB | ~4.3 GB | 1–2 s | + +**Default `qwen2.5:1.5b`**: best demo experience on the cheapest CPU node. + +`STORAGE_GB` for the PVC: at least `model_disk × 2`. 20 Gi covers any single 7B model with room for one alternate. + +## LLM tool-calling reliability — the part that surprises everyone + +The Entra value-prop is the **token chain**, not the LLM. But the same demo UI also exposes an Ollama-driven tool-calling path, and small CPU-only models do not reliably emit `tool_calls`. Customers see this and assume something is wrong with the auth setup. It isn't. + +| Choice | Cost delta | Tool-calling result | +|---|---|---| +| **Default**: `Standard_D2s_v5` + `qwen2.5:1.5b` | $0 | ⚡ Direct works always; 💻 Ollama is "best-effort" — sometimes calls the tool, often skips it and hallucinates | +| Bump nodes to `Standard_D8s_v5` + `qwen2.5:7b` (CPU) | ~+$210/mo | Tool calls reliably, but 15–30 s/turn | +| Add a GPU node pool (`Standard_NC4as_T4_v3`, taint `sku=gpu:NoSchedule`) and pin Ollama there | ~+$500/mo | Reliable AND fast (<5 s/turn) | +| **Recommended for enterprise demos**: replace Ollama with **Azure OpenAI** | ~$10–50/mo pay-per-token | Tool calls reliably; removes the entire Ollama Deployment + PVC; sidecar pattern is unchanged | + +**Practical guidance:** to verify the auth chain in any size demo, use ⚡ Direct mode. To showcase end-to-end LLM-driven tool calling, either pay for a GPU node or swap to Azure OpenAI. + +## ACR + +| `ACR_SKU` | Storage | Geo | When to use | +|---|---|---|---| +| **`Basic`** | 10 GB | single region | Demo, samples, customer adopts → fork | +| `Standard` | 100 GB | single region | If you're storing multiple model-baked images | +| `Premium` | 500 GB | geo-replication | Production multi-region | + +## Ingress + +| `INGRESS_TYPE` | What gets installed | Cost / complexity | +|---|---|---| +| **`LoadBalancer`** | None — just a public Standard LB on the agent Service | Free (LB rule fee ~$18/mo); zero extra components | +| `ingress-nginx` | NGINX ingress controller via Helm | Adds 1 deployment; needs cert-manager for TLS | +| `appgw` | AKS Application Gateway add-on (AGIC) | App Gateway base ~$240/mo; managed TLS via Key Vault | + +Default `LoadBalancer` because the goal is a working demo, not a hardened production gateway. + +## Logs + +| `ENABLE_LOGS` | What you get | Cost | +|---|---|---| +| **`none`** | `kubectl logs` against pods | Free; lost on pod deletion | +| `azure-monitor-container-insights` | Container Insights with retained logs | ~$2–3/GB ingested | + +Default `none` for demo. Flip on once you hit a "why is my pod crashing" moment that requires history. + +## Total estimated demo cost (East US, PAYG, default everything) + +| Item | $/mo | +|---|---| +| 2 × Standard_D2s_v5 | ~$140 | +| ACR Basic | ~$5 | +| Standard LB rule | ~$18 | +| 20 GB managed-csi PVC | ~$1.5 | +| Public IP | ~$3.5 | +| **Total** | **~$170/mo** | + +Stop the cluster (`az aks stop`) when not in use and total drops to ~$30/mo (storage + ACR only). diff --git a/.claude/skills/deploy-agent-aks-dev/references/smoke-test.md b/.claude/skills/deploy-agent-aks-dev/references/smoke-test.md new file mode 100644 index 0000000..ba5fd7c --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/smoke-test.md @@ -0,0 +1,65 @@ +# Local smoke test on `kind` + +**Goal:** validate every manifest and the agent-↔-sidecar wiring on the user's laptop, with no Azure resources, before paying for AKS. + +## What's tested + +| Concern | Covered? | +|---|---| +| YAML parses & passes k8s API validation | ✅ | +| Container images build and run | ✅ | +| Agent reaches sidecar on `localhost:5000` | ✅ | +| Agent reaches `weather-api` Service | ✅ | +| Agent reaches `ollama` Service and model loads | ✅ | +| Sidecar acquires Blueprint token (ClientSecret mode) | ✅ | +| Workload Identity assertion file flow | ❌ (only works on real AKS — uses `ClientSecret` overlay instead) | +| End-to-end OBO with the Client SPA | ❌ (requires a deployed redirect URI — defer to Azure) | + +So the smoke test proves the **kubernetes wiring** is correct. The **secretless credential path** still must be tested on real AKS in Step 7 of the main skill. + +## Requirements + +- Docker Desktop (or any Docker-compatible engine). +- `kind` ≥ 0.20. +- `kubectl` ≥ 1.28. +- A Blueprint client secret (set `BLUEPRINT_CLIENT_SECRET`). For an isolated smoke test where you don't want to use a real tenant, set it to a junk value — the sidecar will fail to mint tokens but every other component should still come up healthy, and `/status` will return `sidecar_reachable: true, token_acquired: false`. + +## Usage + +```bash +source /tmp/deploy-vars.sh # for TENANT_ID, BLUEPRINT_APP_ID, *_CLIENT_ID +export BLUEPRINT_CLIENT_SECRET="" + +bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh +# ... runs for 5-10 min on first run (mostly Ollama model pull) ... +# Last line: SMOKE PASS or SMOKE FAIL: + +# Cleanup: +bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh --cleanup +``` + +## How it differs from production manifests + +The script generates a temp overlay that replaces only the sidecar's credential block: + +```yaml +# overlay applied to 40-agent.yaml only +- name: AzureAd__ClientCredentials__0__SourceType + value: ClientSecret +- name: AzureAd__ClientCredentials__0__ClientSecret + valueFrom: + secretKeyRef: { name: blueprint-secret, key: client-secret } +``` + +…and drops the `azure.workload.identity/use: "true"` label. Everything else — namespace, KSA, Services, PVC, Deployment shapes — is unchanged. + +## Interpreting failures + +| `SMOKE FAIL: <…>` | Most likely cause | +|---|---| +| `kind cluster create` | Docker not running | +| `image load` | `docker build` failed — inspect `kind-build.log` | +| `weather-api rollout` | Port 8080 conflict, or weather-api image broken | +| `ollama rollout` | initContainer hit a network timeout pulling the model; rerun | +| `agent rollout` | Sidecar crash — `kubectl logs deploy/llm-agent -c sidecar` | +| `/status non-200` | Agent can't reach Ollama Service; check DNS in pod | diff --git a/.claude/skills/deploy-agent-aks-dev/references/troubleshooting.md b/.claude/skills/deploy-agent-aks-dev/references/troubleshooting.md new file mode 100644 index 0000000..3d0f900 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/troubleshooting.md @@ -0,0 +1,97 @@ +# Troubleshooting matrix + +Match symptom → cause → fix. Most issues fall into 4 buckets: workload identity wiring, image pull, Ollama, post-deploy Entra config. + +## Quick reference — actual K8s object names + +These names are what the manifests create. Use these in every `kubectl` command: + +| Object | Name | +|---|---| +| Namespace | `agentid` | +| ServiceAccount | `agent-sa` | +| Agent Deployment | `llm-agent` (pod label `app=llm-agent`) | +| Agent containers (in same pod) | `llm-agent` and `sidecar` | +| Other Deployments | `weather-api`, `ollama` | +| LoadBalancer Service (UI) | `llm-agent` | + +Common commands (use these verbatim — `app=agent` or container `auth-sidecar` will silently match nothing): +``` +kubectl get pods -n agentid +kubectl get svc -n agentid llm-agent # external IP +kubectl logs -n agentid -l app=llm-agent -c sidecar --tail=50 # sidecar +kubectl logs -n agentid -l app=llm-agent -c llm-agent --tail=50 # web UI +kubectl exec -n agentid deploy/llm-agent -c sidecar -- env | grep ^AZURE_ +``` + +## Workload Identity / token acquisition + +| Symptom | Cause | Fix | +|---|---|---| +| Sidecar logs: `AADSTS70021: No matching federated identity record` | FIC subject mismatch | Recreate FIC: `subject=system:serviceaccount:agentid:agent-sa` exactly. Spaces or wrong namespace = no match. | +| Sidecar logs: `AADSTS700016: Application not found` | `AzureAd__ClientId` is the Agent ID, not the Blueprint | Set `AzureAd__ClientId=$BLUEPRINT_APP_ID` | +| Sidecar logs: `Could not load file or assembly` / restarts | Wrong image tag | Use `mcr.microsoft.com/entra-sdk/auth-sidecar:1.0.0-azurelinux3.0-distroless` | +| `kubectl exec sidecar -- env \| grep AZURE_` returns nothing | Pod missing label `azure.workload.identity/use: "true"` | Add to pod template, restart deployment | +| `AZURE_FEDERATED_TOKEN_FILE` set but file is empty | KSA missing annotations | Annotate KSA with `azure.workload.identity/client-id` and `tenant-id` | +| Sidecar logs `FileNotFoundException: ...azure-identity-token` | Mutating webhook didn't fire | `az aks update -g $RG -n $AKS --enable-workload-identity`; restart pod | + +## Image pull + +| Symptom | Cause | Fix | +|---|---|---| +| `ImagePullBackOff` on llm-agent or weather-api | ACR not attached to AKS | `az aks update -g $RG -n $AKS --attach-acr $ACR_NAME` | +| `ImagePullBackOff` on `ollama/ollama:latest` | Docker Hub rate limit | Pull-through cache in ACR: `az acr import --source docker.io/ollama/ollama:latest`, change manifest to use ACR copy | + +## Ollama + +| Symptom | Cause | Fix | +|---|---|---| +| Ollama pod crash-loops, OOMKilled | Model too large for node | Switch `OLLAMA_MODEL` to `qwen2.5:1.5b` or bump `NODE_VM_SIZE` | +| `/status` returns `ollama_available: false` | Init container still pulling model | Wait — first pull is 1–5 min | +| Init container hangs on `ollama pull` | Network egress blocked | Check NSG / firewall allows `registry.ollama.ai` | +| Agent gets 500s when asking questions | Wrong model name in `OLLAMA_MODEL` env vs what initContainer pulled | Make sure they match exactly (`qwen2.5:1.5b` ≠ `qwen2.5`) | + +## Post-deploy Entra config + +| Symptom | Cause | Fix | +|---|---|---| +| OBO sign-in fails with `AADSTS65001` | Agent → Graph User.Read not admin-consented | Run `scripts/grant-agent-obo-consent.ps1` (AKS-local copy) | +| OBO sign-in fails with `AADSTS50011: redirect URI mismatch` | Agent FQDN not added to SPA app | Run `scripts/add-spa-redirect-uri.sh` with `APP_FQDN=` | +| OBO sign-in works but agent can't call Graph as user | Blueprint not configured for OBO | Re-run `scripts/setup-obo-blueprint-for-aks.ps1` | +| weather-api returns 401 to agent | `TENANT_ID` env on weather-api wrong, or `appid` in token doesn't match Agent ID | Confirm both containers see the same `TENANT_ID`; check `kubectl logs deploy/weather-api` for the validation error | +| Sign-in popup throws `pkce_not_created: TypeError: Cannot read properties of undefined (reading 'subtle')` | MSAL needs `window.crypto.subtle`, which browsers gate on **secure context**. `http://` is not a secure context; `http://localhost:*` is exempt. | Run `scripts/port-forward.sh` and use `http://localhost:8080` for sign-in. Production-style fix: front the Service with HTTPS (cert-manager + NGINX, or AGIC + Key Vault). | + +## Cross-tenant federation + +| Symptom | Cause | Fix | +|---|---|---| +| `03-federate-blueprint.ps1` fails with `Authorization_RequestDenied` | `Connect-MgGraph` ran against the wrong tenant (the Azure-sub tenant, not the Entra tenant where the Blueprint lives) | Re-run with explicit `-TenantId $TENANT_ID` (the Entra/Blueprint tenant), independent of `SUBSCRIPTION_TENANT_ID` | +| `az aks ...` works but Graph calls 401 | Single `az login` only covered one tenant; CLI cached the wrong context for Graph | `az login --tenant $TENANT_ID` once, then `az login --tenant $SUBSCRIPTION_TENANT_ID` and `az account set --subscription $SUBSCRIPTION_ID`. The two tokens live side-by-side. | +| Sidecar logs `AADSTS70021` even though FIC was created | FIC was added on the Blueprint **in the Azure-sub tenant**, not the Entra tenant | Delete the wrong FIC. Recreate it on the Blueprint app in the Entra tenant (`TENANT_ID`). | +| Pod env shows `AZURE_TENANT_ID=$SUBSCRIPTION_TENANT_ID` | `40-agent.yaml` rendered before `TENANT_ID` was the Entra tenant | Re-render manifests with `TENANT_ID` set to the Entra/Blueprint tenant, `kubectl apply`, restart pod | + +See [`cross-tenant-federation.md`](./cross-tenant-federation.md) for the full pattern. + +## Manifest rendering (`envsubst`) + +| Symptom | Cause | Fix | +|---|---|---| +| Rendered YAML still contains `$TENANT_ID` literal | `envsubst` without an explicit var list substitutes **only exported** vars; if you forgot to `source /tmp/deploy-vars.sh` or used `set` (not `export`), nothing happens | `set -a; source /tmp/deploy-vars.sh; set +a` so all assignments are auto-exported | +| Rendered YAML has empty strings where vars should be | Variable was sourced but had a blank value, or shell variable shadowed it | `echo "TENANT_ID=$TENANT_ID"` before rendering. Prefer the explicit-varlist form: `envsubst '$TENANT_ID $BLUEPRINT_APP_ID ...' < file.yaml` to fail loudly on typos | +| `envsubst: command not found` (Windows / Git Bash) | `gettext` not installed | Git Bash ships it under `/usr/bin/envsubst.exe`; otherwise `winget install GnuWin32.Gettext` or `choco install gettext` | + +## Networking + +| Symptom | Cause | Fix | +|---|---|---| +| LB IP stays `` for > 5 min | Subscription LB quota exhausted or policy blocks public IPs | Switch to `INGRESS_TYPE=ingress-nginx` | +| Agent can resolve `weather-api` but gets connection refused | CoreDNS cache stale or pod still starting | `kubectl rollout status deploy/weather-api`; restart agent | +| `kubectl port-forward` works but LB doesn't | NSG on the AKS node subnet blocks 80 | Inspect AKS node subnet NSG rules | + +## Smoke test (kind) + +| Symptom | Cause | Fix | +|---|---|---| +| `kind create cluster` fails with `cgroup` errors | Docker Desktop cgroup v1 incompatibility | Update Docker Desktop ≥ 4.20 | +| `kind load docker-image` slow / hangs | Large image transfer over Docker socket | Be patient (3–5 min for ollama image); or use a kind config with local registry | +| Sidecar in ClientSecret mode logs `AADSTS7000215: Invalid client secret` | Junk secret used | Replace with real Blueprint secret, or accept this and only verify non-token paths | diff --git a/.claude/skills/deploy-agent-aks-dev/references/workload-identity.md b/.claude/skills/deploy-agent-aks-dev/references/workload-identity.md new file mode 100644 index 0000000..1883134 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/workload-identity.md @@ -0,0 +1,80 @@ +# Workload Identity deep-dive + +## Why this works without a UAMI + +The auth-sidecar (`Microsoft.Identity.Web`) supports several `ClientCredentials` sources. Two are relevant: + +| `SourceType` | What it does | Where it makes sense | +|---|---|---| +| `SignedAssertionFromManagedIdentity` | Calls IMDS to get a JWT signed by the MI; uses that as the federated client assertion | ACA, App Service, VMs (real MI) | +| `SignedAssertionFilePath` | Reads a JWT directly from a file on disk and uses it as the assertion | **AKS with Workload Identity** | + +On AKS with `--enable-workload-identity`, the mutating webhook (triggered by the pod label `azure.workload.identity/use: "true"`) does two things: + +1. Injects env vars `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_FEDERATED_TOKEN_FILE`, `AZURE_AUTHORITY_HOST`. +2. Projects a service account token at `/var/run/secrets/azure/tokens/azure-identity-token`, signed by the **AKS cluster's OIDC issuer**, with audience `api://AzureADTokenExchange`. + +That projected token IS already a valid federated assertion for any Entra app that trusts that issuer + subject. So we point the sidecar at it directly with `SignedAssertionFilePath` and skip the UAMI hop entirely. + +## The FIC contract + +The federated identity credential on the Blueprint app says: + +``` +issuer = # e.g. https://eastus.oic.prod-aks.azure.com/// +subject = system:serviceaccount:agentid:agent-sa +audiences = [ api://AzureADTokenExchange ] +``` + +The sidecar sends `client_assertion=` to `login.microsoftonline.com`, Entra validates the signature against the AKS OIDC keys, checks issuer+subject+audience, and mints a Blueprint token. The Blueprint token is then used (via OBO or client-credentials) to mint the **Agent Identity** token for the downstream API. + +## Token rotation + +| Token | Lifetime | Refreshed by | +|---|---|---| +| Projected SA token (assertion) | ~1 h | Workload identity webhook (re-writes the file ~10 min before expiry) | +| Blueprint access token | ~1 h | Sidecar (Microsoft.Identity.Web cache) | +| Agent Identity token | ~1 h | Sidecar on each `GetAuthorizationHeader` call (cached) | + +Because `SignedAssertionFilePath` re-reads the file on every assertion request, rotation is automatic. Nothing to configure. + +## Why not just use UAMI + `SignedAssertionFromManagedIdentity`? + +It also works: +- KSA federated to UAMI (standard AKS workload identity pattern). +- Blueprint federated to UAMI (FIC subject = UAMI objectId). +- Sidecar with `SignedAssertionFromManagedIdentity`. + +But this introduces: +- An extra Azure resource (the UAMI) to provision, manage RBAC on, and clean up. +- A second federation hop (KSA→UAMI→Blueprint instead of KSA→Blueprint). +- An IMDS-style round-trip in the sidecar on every token request. + +Direct KSA→Blueprint is one fewer resource, one fewer hop, same security posture. Recommended for new deployments. If your organization standardizes on UAMI-per-workload for IAM auditing, switch the sidecar env to `SignedAssertionFromManagedIdentity` and add the UAMI hop — manifests stay otherwise identical. + +## Validating workload identity is wired correctly + +```bash +# Token file exists? +kubectl -n agentid exec deploy/llm-agent -c sidecar -- \ + ls -l /var/run/secrets/azure/tokens/ + +# Env vars injected? +kubectl -n agentid exec deploy/llm-agent -c sidecar -- env | grep AZURE_ + +# Decode the assertion (audience + iss + sub) +kubectl -n agentid exec deploy/llm-agent -c sidecar -- \ + sh -c 'cat /var/run/secrets/azure/tokens/azure-identity-token' \ + | cut -d. -f2 | base64 -d 2>/dev/null +``` + +Expected: `iss` = AKS OIDC URL, `sub` = `system:serviceaccount:agentid:agent-sa`, `aud` = `api://AzureADTokenExchange`. + +## Common pitfalls + +| Symptom | Cause | Fix | +|---|---|---| +| Sidecar logs `AADSTS70021: No matching federated identity record found` | FIC subject doesn't match the projected token's `sub` | Recreate the FIC with `subject = system:serviceaccount::` exactly | +| `AZURE_FEDERATED_TOKEN_FILE` env not set in sidecar | Pod missing `azure.workload.identity/use: "true"` label | Add the label to **the pod template**, not the Deployment | +| Token file empty / 404 from Entra | KSA missing `azure.workload.identity/client-id` annotation | Annotate the KSA with the Blueprint app's client ID | +| `kubectl get pod` shows no `AZURE_*` env | Workload identity webhook not installed | `az aks update --enable-workload-identity` on the cluster | diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/01-create-aks.sh b/.claude/skills/deploy-agent-aks-dev/scripts/01-create-aks.sh new file mode 100644 index 0000000..f20f4c2 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/01-create-aks.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Create resource group, ACR, and an AKS cluster with OIDC issuer + +# Azure Workload Identity enabled, then attach ACR. +set -euo pipefail + +: "${TENANT_ID:?}"; : "${SUBSCRIPTION_ID:?}"; : "${RG:?}"; : "${LOCATION:?}" +: "${AKS_NAME:?}"; : "${ACR_NAME:?}"; : "${NODE_COUNT:?}"; : "${NODE_VM_SIZE:?}" +export SUBSCRIPTION_TENANT_ID="${SUBSCRIPTION_TENANT_ID:-$TENANT_ID}" + +# Verify az CLI is authenticated to the tenant that owns the subscription. +CURRENT_TENANT=$(az account show --query tenantId -o tsv 2>/dev/null || true) +if [[ "$CURRENT_TENANT" != "$SUBSCRIPTION_TENANT_ID" ]]; then + echo "ERROR: az CLI is signed into tenant '$CURRENT_TENANT' but the target" + echo " subscription lives in tenant '$SUBSCRIPTION_TENANT_ID'." + echo " Run: az login --tenant $SUBSCRIPTION_TENANT_ID" >&2 + exit 1 +fi + +az account set --subscription "$SUBSCRIPTION_ID" + +echo "[1/4] Resource group" +az group create -n "$RG" -l "$LOCATION" -o none + +echo "[2/4] ACR ($ACR_NAME)" +az acr create -g "$RG" -n "$ACR_NAME" --sku Basic --admin-enabled false -o none 2>/dev/null || true + +echo "[3/4] AKS cluster ($AKS_NAME) — OIDC + Workload Identity" +az aks create \ + -g "$RG" -n "$AKS_NAME" \ + --location "$LOCATION" \ + --node-count "$NODE_COUNT" \ + --node-vm-size "$NODE_VM_SIZE" \ + --enable-oidc-issuer \ + --enable-workload-identity \ + --enable-managed-identity \ + --generate-ssh-keys \ + -o none + +echo "[4/4] Attach ACR to AKS (grants kubelet AcrPull)" +az aks update -g "$RG" -n "$AKS_NAME" --attach-acr "$ACR_NAME" -o none + +OIDC_ISSUER=$(az aks show -g "$RG" -n "$AKS_NAME" --query "oidcIssuerProfile.issuerUrl" -o tsv) +echo "export OIDC_ISSUER=\"$OIDC_ISSUER\"" >> "${VARS_FILE:-/tmp/deploy-vars.sh}" +echo "OIDC issuer: $OIDC_ISSUER" + +az aks get-credentials -g "$RG" -n "$AKS_NAME" --overwrite-existing +kubectl get nodes diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh b/.claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh new file mode 100644 index 0000000..d91a46a --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Build & push llm-agent and weather-api to ACR using `az acr build` +# (no local Docker required). Ollama uses the upstream image as-is — the +# 30-ollama.yaml manifest pulls the model into a PVC via initContainer. +set -euo pipefail +: "${ACR_NAME:?}" + +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +# Locate the sidecar source dirs (`dev/` and `weather-api/`). Two supported layouts: +# A) Upstream repo layout: /sidecar/{dev,weather-api,aks} (when this +# skill is shipped inside microsoft/entra-agentid-samples) +# B) Reference-clone layout: /reference/repo/sidecar/{dev,weather-api} +# (when developing this skill outside the upstream repo) +find_sidecar_root() { + local candidate + for candidate in \ + "$SCRIPT_DIR/../../../../sidecar" \ + "$SCRIPT_DIR/../../../sidecar" \ + "$SCRIPT_DIR/../../../reference/repo/sidecar" ; do + if [[ -d "$candidate/dev" && -d "$candidate/weather-api" ]]; then + ( cd "$candidate" && pwd ); return 0 + fi + done + return 1 +} +SIDECAR_ROOT="$( find_sidecar_root )" || { + echo "ERROR: could not locate sidecar/{dev,weather-api}. Tried upstream and reference layouts." >&2 + exit 1 +} + +echo "[1/2] llm-agent (source: $SIDECAR_ROOT/dev)" +az acr build --registry "$ACR_NAME" \ + --image agent-id-dev/llm-agent:1.0.0 \ + --platform linux/amd64 \ + "$SIDECAR_ROOT/dev" + +echo "[2/2] weather-api (source: $SIDECAR_ROOT/weather-api)" +az acr build --registry "$ACR_NAME" \ + --image agent-id-dev/weather-api:1.0.0 \ + --platform linux/amd64 \ + "$SIDECAR_ROOT/weather-api" diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/03-federate-blueprint.ps1 b/.claude/skills/deploy-agent-aks-dev/scripts/03-federate-blueprint.ps1 new file mode 100644 index 0000000..cac358a --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/03-federate-blueprint.ps1 @@ -0,0 +1,42 @@ +# Federate the AKS KSA directly to the Blueprint app. +# Subject = system:serviceaccount:agentid:agent-sa, Audience = api://AzureADTokenExchange. +# This is the only federation chain — no UAMI in the middle. + +param( + [Parameter(Mandatory=$true)] [string] $TenantId, + [Parameter(Mandatory=$true)] [string] $BlueprintAppId, + [Parameter(Mandatory=$true)] [string] $OidcIssuerUrl, + [string] $Namespace = "agentid", + [string] $ServiceAccount = "agent-sa", + [string] $FicName = "aks-agent-sa" +) + +$ErrorActionPreference = "Stop" + +Import-Module Microsoft.Graph.Authentication -ErrorAction Stop +Connect-MgGraph -TenantId $TenantId -Scopes "AgentIdentityBlueprint.AddRemoveCreds.All" -NoWelcome | Out-Null + +$subject = "system:serviceaccount:$Namespace`:$ServiceAccount" +$body = @{ + name = $FicName + issuer = $OidcIssuerUrl + subject = $subject + audiences = @("api://AzureADTokenExchange") + description = "AKS KSA $subject" +} | ConvertTo-Json -Depth 5 + +$uri = "https://graph.microsoft.com/beta/applications(appId='$BlueprintAppId')/federatedIdentityCredentials" + +try { + Invoke-MgGraphRequest -Method POST -Uri $uri -Body $body -ContentType "application/json" | Out-Null + Write-Host "Federated credential '$FicName' created on Blueprint $BlueprintAppId" + Write-Host " issuer : $OidcIssuerUrl" + Write-Host " subject : $subject" +} +catch { + if ($_.Exception.Message -match "already exists|FederatedIdentityCredential with the same") { + Write-Host "Federated credential '$FicName' already exists — skipping." + } else { + throw + } +} diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/04-apply-manifests.sh b/.claude/skills/deploy-agent-aks-dev/scripts/04-apply-manifests.sh new file mode 100644 index 0000000..a3406d1 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/04-apply-manifests.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Render manifests with envsubst and apply. +set -euo pipefail + +for v in TENANT_ID BLUEPRINT_APP_ID AGENT_CLIENT_ID ACR_NAME OLLAMA_MODEL; do + if [[ -z "${!v:-}" ]]; then echo "Missing \$$v" >&2; exit 1; fi +done + +# CLIENT_SPA_APP_ID is OPTIONAL. The default AKS path is autonomous (app-only) +# auth via Workload Identity — no user MSAL sign-in. Only set it if you also +# want to enable user-OBO mode in the llm-agent UI (mirrors ACA dev skill). +export CLIENT_SPA_APP_ID="${CLIENT_SPA_APP_ID:-not-used}" + +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +MANIFESTS="$SCRIPT_DIR/../manifests" +OUT="/tmp/agentid-aks-rendered" +mkdir -p "$OUT" + +if ! command -v envsubst >/dev/null; then + echo "envsubst not found. apt: gettext-base | brew: gettext | choco: gettext" >&2 + exit 1 +fi + +for f in "$MANIFESTS"/*.yaml; do + envsubst '$TENANT_ID $BLUEPRINT_APP_ID $AGENT_CLIENT_ID $ACR_NAME $OLLAMA_MODEL $CLIENT_SPA_APP_ID' < "$f" > "$OUT/$(basename "$f")" +done + +kubectl apply -f "$OUT/00-namespace.yaml" +kubectl apply -f "$OUT/10-serviceaccount.yaml" +kubectl apply -f "$OUT/20-weather-api.yaml" +kubectl apply -f "$OUT/30-ollama.yaml" +kubectl apply -f "$OUT/40-agent.yaml" +kubectl apply -f "$OUT/50-ingress.yaml" + +echo +echo "Waiting for rollouts..." +kubectl -n agentid rollout status deploy/weather-api --timeout=180s +kubectl -n agentid rollout status deploy/ollama --timeout=600s +kubectl -n agentid rollout status deploy/llm-agent --timeout=180s + +echo +echo "Waiting for LoadBalancer IP..." +for i in $(seq 1 60); do + IP=$(kubectl -n agentid get svc llm-agent -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true) + [[ -n "$IP" ]] && break + sleep 5 +done +echo "Agent UI: http://${IP:-}/" diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh b/.claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh new file mode 100644 index 0000000..810ef60 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# add-spa-redirect-uri.sh — register browser sign-in URIs on the Client SPA app. +# +# AKS-specific behavior: +# 1. ALWAYS registers `http://localhost:8080/`. The LoadBalancer exposes the +# agent over plain HTTP, and raw-IP HTTP is NOT a browser "secure context" +# — so MSAL.js (which relies on Web Crypto / PKCE) refuses to open the +# sign-in popup. Loopback IS a secure context, so OBO works via +# `kubectl port-forward svc/llm-agent 8080:80` (see scripts/port-forward.sh). +# 2. If APP_FQDN is set, ALSO registers `${REDIRECT_SCHEME:-http}://$APP_FQDN/` +# so the LoadBalancer IP works for autonomous-mode browsing (no sign-in). +# 3. If REDIRECT_URI is set, registers that string verbatim instead. +# +# Cross-tenant aware: TENANT_ID is the Entra tenant where the SPA lives. The +# script asks `az` for a Graph token in that tenant explicitly so it works even +# when the active `az account` is in a different (subscription) tenant. +# +# Idempotent — fetches existing spa.redirectUris first, only PATCHes the +# difference. +# +# Required env (from /tmp/deploy-vars.sh): +# CLIENT_SPA_APP_ID +# TENANT_ID +# Optional: +# APP_FQDN (LoadBalancer IP or DNS; will be wrapped with scheme) +# REDIRECT_SCHEME (http|https, default http for AKS) +# REDIRECT_URI (overrides APP_FQDN + scheme; verbatim) +# PORT_FORWARD_URI (default http://localhost:8080/; set empty to skip) + +set -euo pipefail + +: "${CLIENT_SPA_APP_ID:?CLIENT_SPA_APP_ID required in env (e.g. /tmp/deploy-vars.sh)}" +: "${TENANT_ID:?TENANT_ID required (Entra tenant where the SPA lives)}" + +PORT_FORWARD_URI="${PORT_FORWARD_URI-http://localhost:8080/}" +APP_FQDN="${APP_FQDN:-}" +REDIRECT_SCHEME="${REDIRECT_SCHEME:-http}" +REDIRECT_URI="${REDIRECT_URI:-}" + +if [[ -z "$REDIRECT_URI" && -n "$APP_FQDN" ]]; then + REDIRECT_URI="${REDIRECT_SCHEME}://${APP_FQDN}/" +fi + +TOK=$(az account get-access-token --tenant "$TENANT_ID" \ + --resource https://graph.microsoft.com --query accessToken -o tsv 2>/dev/null \ + || az account get-access-token --resource https://graph.microsoft.com --query accessToken -o tsv) +[[ -z "$TOK" ]] && { echo "ERROR: failed to acquire Graph token for tenant $TENANT_ID" >&2; exit 1; } + +EXISTING=$(curl -sS --fail -H "Authorization: Bearer $TOK" \ + "https://graph.microsoft.com/v1.0/applications(appId='$CLIENT_SPA_APP_ID')?\$select=id,spa") + +BODY=$(python3 - </dev/null + +echo "Added SPA redirect URIs: $ADDED" diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/deploy-aks-dev.sh b/.claude/skills/deploy-agent-aks-dev/scripts/deploy-aks-dev.sh new file mode 100644 index 0000000..14d1bbf --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/deploy-aks-dev.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# One-shot orchestrator. Source /tmp/deploy-vars.sh first. +# +# cp scripts/deploy-vars.sh.template /tmp/deploy-vars.sh +# # edit /tmp/deploy-vars.sh +# source /tmp/deploy-vars.sh +# bash scripts/deploy-aks-dev.sh +# +# Prerequisite: Entra Agent ID Blueprint + Agent already exist. Set BLUEPRINT_APP_ID +# and AGENT_CLIENT_ID in deploy-vars.sh. CLIENT_SPA_APP_ID is OPTIONAL (autonomous +# path doesn't need it). +set -euo pipefail + +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +export VARS_FILE="${VARS_FILE:-/tmp/deploy-vars.sh}" + +for v in TENANT_ID SUBSCRIPTION_ID RG LOCATION AKS_NAME ACR_NAME \ + NODE_COUNT NODE_VM_SIZE BLUEPRINT_APP_ID AGENT_CLIENT_ID \ + OLLAMA_MODEL; do + if [[ -z "${!v:-}" ]]; then + echo "ERROR: \$$v unset. Source $VARS_FILE." >&2; exit 1 + fi +done +export CLIENT_SPA_APP_ID="${CLIENT_SPA_APP_ID:-not-used}" +# Defaults to TENANT_ID (single-tenant deploy). Override only for cross-tenant. +export SUBSCRIPTION_TENANT_ID="${SUBSCRIPTION_TENANT_ID:-$TENANT_ID}" + +echo "============================================================" +echo " AKS deploy plan" +if [[ "$SUBSCRIPTION_TENANT_ID" != "$TENANT_ID" ]]; then + echo " *** CROSS-TENANT DEPLOY ***" + echo " Entra tenant (Blueprint/Agent) : $TENANT_ID" + echo " Azure sub tenant (AKS/ACR) : $SUBSCRIPTION_TENANT_ID" + echo " Subscription : $SUBSCRIPTION_ID" +else + echo " Tenant/Sub : $TENANT_ID / $SUBSCRIPTION_ID" +fi +echo " RG/Location: $RG / $LOCATION" +echo " AKS / ACR : $AKS_NAME / $ACR_NAME" +echo " Nodes : $NODE_COUNT × $NODE_VM_SIZE" +echo " Model : $OLLAMA_MODEL" +echo "============================================================" + +bash "$SCRIPT_DIR/01-create-aks.sh" +# 01 appends OIDC_ISSUER to VARS_FILE — pick it up. +# shellcheck disable=SC1090 +source "$VARS_FILE" + +bash "$SCRIPT_DIR/02-build-and-push.sh" + +pwsh -NoProfile -File "$SCRIPT_DIR/03-federate-blueprint.ps1" \ + -TenantId "$TENANT_ID" \ + -BlueprintAppId "$BLUEPRINT_APP_ID" \ + -OidcIssuerUrl "$OIDC_ISSUER" \ + -FicName "${FIC_NAME:-aks-agent-sa}" + +bash "$SCRIPT_DIR/04-apply-manifests.sh" + +echo +echo "============================================================" +LB_IP=$(kubectl get svc -n agentid llm-agent -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true) +echo " Done. Agent UI: http://${LB_IP:-}" +echo +echo " Verify:" +echo " kubectl get pods -n agentid" +echo " kubectl logs -n agentid -l app=llm-agent -c sidecar --tail=50" +echo +echo " Autonomous (app-only) path is ready as-is — no Entra config needed." +echo " Open the URL above and chat; the agent acquires its own Agent Identity" +echo " token via the Blueprint FIC." +echo +echo " For user On-Behalf-Of (sign-in) mode (REQUIRED for the 'Sign In' button):" +echo " 1) Register SPA redirect URIs (localhost:8080 for OBO + LB-IP for autonomous):" +echo " APP_FQDN=\"\$LB_IP\" bash \"\$SCRIPT_DIR/add-spa-redirect-uri.sh\"" +echo " 2) Grant Agent -> Graph delegated User.Read admin consent:" +echo " pwsh -NoProfile -File \"\$SCRIPT_DIR/grant-agent-obo-consent.ps1\" \\" +echo " -AgentAppId \"$AGENT_CLIENT_ID\" -TenantId \"$TENANT_ID\"" +echo " 3) Port-forward to localhost (PKCE needs a secure context — raw HTTP IPs" +echo " are not secure-context; loopback is exempt):" +echo " bash \"\$SCRIPT_DIR/port-forward.sh\"" +echo " Then open http://localhost:8080 and click Sign In." +echo "============================================================" diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/deploy-vars.sh.template b/.claude/skills/deploy-agent-aks-dev/scripts/deploy-vars.sh.template new file mode 100644 index 0000000..8b23933 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/deploy-vars.sh.template @@ -0,0 +1,105 @@ +# Copy to /tmp/deploy-vars.sh, fill in values, then `source /tmp/deploy-vars.sh`. +# Scripts in this skill append newly-discovered IDs (OIDC_ISSUER, APP_FQDN) back +# to this file as they run. Re-source after each step that prints "appended:". + +# ============================================================================ +# Entra tenant — where the Blueprint, Agent Identity, and Client SPA live. +# This is the tenant whose admin ran Start-EntraAgentIDWorkflow. +# ============================================================================ +export TENANT_ID="" + +# ============================================================================ +# Azure subscription tenant — where the AKS cluster, ACR, and RG live. +# OPTIONAL: defaults to $TENANT_ID for single-tenant deployments. +# Set this only when the Azure sub lives in a DIFFERENT tenant than the +# Blueprint (cross-tenant federation — supported, requires two `az login`s). +# See references/cross-tenant-federation.md. +# ============================================================================ +export SUBSCRIPTION_TENANT_ID="${SUBSCRIPTION_TENANT_ID:-$TENANT_ID}" + +# ============================================================================ +# Azure subscription / location / RG +# ============================================================================ +export SUBSCRIPTION_ID="" +export RG="rg-agentid-aks-dev" +export LOCATION="eastus2" # any region with AKS + ACR + your VM SKU available + +# ============================================================================ +# AKS + ACR +# ============================================================================ +export AKS_NAME="aks-agentid-dev" + +# ACR name: 5–50 chars, lowercase alphanumeric, GLOBALLY unique. Suggestion: +# ACR_NAME="acragentid$(openssl rand -hex 4)" # compute ONCE, paste here. +# Do NOT use $(...) directly in this file — the file is sourced multiple times +# and each source would regenerate the suffix, drifting between steps. +export ACR_NAME="" + +# ACR SKU: Basic (~$5/mo, 10 GB) | Standard (~$20/mo, 100 GB) | Premium (geo-replication) +# Basic is fine for demos; bumps to Standard if storing >5 baked-Ollama variants. +export ACR_SKU="Basic" + +# Node pool. See references/sku-sizing.md for the full matrix. +# B2s (2 vCPU / 4 GB, ~$30/mo) — minimum; qwen2.5:1.5b only, slow +# D2s_v5 (2 vCPU / 8 GB, ~$70/mo) — comfortable for qwen2.5:1.5b +# D4s_v5 (4 vCPU / 16 GB, ~$140/mo) — required for qwen2.5:7b on CPU +# NC4as_T4_v3 (4 vCPU/28 GB + T4 GPU, ~$540/mo) — qwen2.5:7b at usable latency +# Note: D2s_v5 is unavailable in some subs/regions — fall back to D2s_v3. +export NODE_VM_SIZE="Standard_D4s_v5" +export NODE_COUNT=2 + +# ============================================================================ +# Storage (PVC for Ollama models) +# ============================================================================ +# At least model_disk × 2. 20 GiB covers any single 7B model with room for one +# alternate. Disk grows as you pull more models; never shrinks. +export STORAGE_GB="20" + +# ============================================================================ +# Ingress +# ============================================================================ +# LoadBalancer (default — public Standard LB, ~$18/mo, plain HTTP) +# ingress-nginx (you install Helm chart + cert-manager for TLS) +# appgw (AKS Application Gateway add-on; managed TLS, +$240/mo base) +export INGRESS_TYPE="LoadBalancer" + +# ============================================================================ +# Logs +# ============================================================================ +# none — kubectl logs only; free; lost on pod delete +# azure-monitor-container-insights — retained logs; ~$2-3/GB ingested +export ENABLE_LOGS="none" + +# ============================================================================ +# Entra Agent ID objects — populated by entra-agent-id-setup (Step 1) +# ============================================================================ +export BLUEPRINT_APP_ID="" +export AGENT_CLIENT_ID="" +export CLIENT_SPA_APP_ID="" # OPTIONAL: only set for user-OBO mode + +# Only needed for the `kind` local smoke test (which uses ClientSecret instead +# of Workload Identity). The AKS deployment path never uses this. +export BLUEPRINT_CLIENT_SECRET="" + +# ============================================================================ +# Ollama (only used by sample agent — ignore if bringing your own agent) +# ============================================================================ +# qwen2.5:0.5b ~700 MB — fastest, tool-calling unreliable +# qwen2.5:1.5b ~1.3 GB — DEFAULT; best demo-quality on CPU +# qwen2.5:3b ~2.0 GB — needs D4s+ +# qwen2.5:7b ~4.3 GB — needs D4s+ on CPU, GPU node for usable latency +# llama3.2:1b ~1.3 GB — alternate small model +export OLLAMA_MODEL="qwen2.5:1.5b" + +# ============================================================================ +# FIC naming — only customize when redeploying onto a cluster whose previous +# FIC name is still on the Blueprint. Default `aks-agent-sa` is fine for +# first-time deploys. Set e.g. `aks-agent-sa-v2` on subsequent attempts. +# ============================================================================ +export FIC_NAME="${FIC_NAME:-aks-agent-sa}" + +# ============================================================================ +# Auto-filled by scripts — do not edit by hand +# ============================================================================ +export OIDC_ISSUER="" # filled by 01-create-aks.sh +export APP_FQDN="" # filled by 04-apply-manifests.sh (= LB IP) diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 b/.claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 new file mode 100644 index 0000000..aca5b3b --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 @@ -0,0 +1,42 @@ +# grant-agent-obo-consent.ps1 +# Grant the Agent service principal the delegated Graph permission `User.Read` +# at admin-consent level. Fixes AADSTS65001 on the OBO sign-in flow. +# +# This is the AKS copy of grant-agent-obo-consent.ps1 — content is identical to +# the ACA version because the Entra/Graph operation is k8s-agnostic. It lives +# here too so the AKS skill is self-contained and the PR ships it alongside the +# manifests in this skill's manifests/ directory. +# +# Cross-tenant: -TenantId is the Entra tenant where the Agent app lives (which +# can differ from the Azure subscription tenant). +param( + [Parameter(Mandatory=$true)][string]$AgentAppId, + [Parameter(Mandatory=$true)][string]$TenantId +) +$ErrorActionPreference='Stop' +Connect-MgGraph -Scopes 'AppRoleAssignment.ReadWrite.All','DelegatedPermissionGrant.ReadWrite.All','Application.Read.All','Directory.Read.All' -TenantId $TenantId -NoWelcome | Out-Null + +$agentSp = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/v1.0/servicePrincipals(appId='$AgentAppId')?`$select=id,displayName" +Write-Host "Agent SP: $($agentSp.id) ($($agentSp.displayName))" + +$graphSp = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/v1.0/servicePrincipals(appId='00000003-0000-0000-c000-000000000000')?`$select=id" +Write-Host "Graph SP: $($graphSp.id)" + +$existing = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/v1.0/oauth2PermissionGrants?`$filter=clientId eq '$($agentSp.id)' and resourceId eq '$($graphSp.id)'" +Write-Host "Existing grants: $($existing.value.Count)" +$existing.value | ForEach-Object { Write-Host " scope='$($_.scope)' consentType=$($_.consentType)" } + +$hasUserRead = $existing.value | Where-Object { $_.scope -match 'User\.Read' } | Select-Object -First 1 +if ($hasUserRead) { + Write-Host "User.Read already granted ($($hasUserRead.scope)). Nothing to do." + return +} + +$body = @{ + clientId = $agentSp.id + consentType = 'AllPrincipals' + resourceId = $graphSp.id + scope = 'User.Read' +} | ConvertTo-Json +$r = Invoke-MgGraphRequest -Method POST -Uri 'https://graph.microsoft.com/v1.0/oauth2PermissionGrants' -Body $body -ContentType 'application/json' +Write-Host "Granted. id=$($r.id) scope='$($r.scope)'" diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh b/.claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh new file mode 100644 index 0000000..035b8fb --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# port-forward.sh — open a loopback tunnel to the llm-agent Service so the +# browser-based OBO sign-in flow works. +# +# Why this exists (AKS-specific): +# The LoadBalancer Service in 50-ingress.yaml exposes the agent over plain +# HTTP on a public IP. Browsers do NOT treat http:// as a "secure +# context", so MSAL.js's PKCE flow (which needs Web Crypto's +# `crypto.subtle`) is blocked and the sign-in popup never opens. Loopback +# addresses (localhost, 127.0.0.1) ARE secure-context exempt, so port- +# forwarding the same Service to localhost makes OBO work without setting +# up TLS / cert-manager / DNS. +# +# Direct (autonomous) mode does NOT require OBO and works fine on the raw +# LB IP. This script is only needed to exercise the "Sign In" button. +# +# Companion: scripts/add-spa-redirect-uri.sh registers `http://localhost:8080/` +# as a SPA redirect URI by default for exactly this flow. +# +# Usage: +# bash port-forward.sh # foreground, Ctrl-C to stop +# LOCAL_PORT=9090 bash port-forward.sh +# +# Env (optional): +# NAMESPACE default: agentid +# SERVICE default: llm-agent +# LOCAL_PORT default: 8080 +# REMOTE_PORT default: 80 + +set -euo pipefail + +NAMESPACE="${NAMESPACE:-agentid}" +SERVICE="${SERVICE:-llm-agent}" +LOCAL_PORT="${LOCAL_PORT:-8080}" +REMOTE_PORT="${REMOTE_PORT:-80}" + +echo "Port-forward: http://localhost:${LOCAL_PORT} -> svc/${SERVICE}:${REMOTE_PORT} (ns ${NAMESPACE})" +echo "Open this URL in a browser to use OBO sign-in. Ctrl-C to stop." +echo "" + +kubectl -n "$NAMESPACE" port-forward "svc/${SERVICE}" "${LOCAL_PORT}:${REMOTE_PORT}" diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 b/.claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 new file mode 100644 index 0000000..50e012a --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 @@ -0,0 +1,83 @@ +# setup-obo-blueprint-for-aks.ps1 +# Configure the Blueprint app for the OBO (User.signedIn -> Agent) flow. +# This is the AKS copy of setup-obo-blueprint-for-aca.ps1 — content is identical +# because the Entra/Graph configuration is k8s-agnostic. The only reason it +# lives here too is so the AKS skill is self-contained and the PR ships the +# scripts alongside the manifests in this skill. +# +# What it does (all idempotent): +# A. Set identifierUris = api:// (Blueprint app) +# B. Add OAuth2 scope `access_as_user` (Blueprint app) +# C. Verify result +# D. Add Client SPA requiredResourceAccess: Blueprint/access_as_user +# E. Pre-grant admin consent (oauth2PermissionGrant) (Client SPA -> Blueprint) +# +# Cross-tenant: pass -TenantId for the Entra tenant where the Blueprint + SPA +# live (which can differ from the Azure subscription tenant — see +# references/cross-tenant-federation.md). +param( + [Parameter(Mandatory=$true)][string]$BlueprintAppId, + [Parameter(Mandatory=$true)][string]$TenantId, + [Parameter(Mandatory=$true)][string]$ClientSpaAppId, + [Parameter(Mandatory=$true)][string]$AgentAppId +) +$ErrorActionPreference='Stop' + +Connect-MgGraph -Scopes 'AgentIdentityBlueprint.ReadWrite.All','Application.ReadWrite.All','AgentIdentityBlueprint.AddRemoveCreds.All','AgentIdentityBlueprint.Create','DelegatedPermissionGrant.ReadWrite.All','Application.Read.All','AgentIdentityBlueprintPrincipal.Create','AppRoleAssignment.ReadWrite.All','Directory.Read.All','User.Read' -TenantId $TenantId -NoWelcome | Out-Null + +$appIdUri = "api://$BlueprintAppId" +$scopeId = [guid]::NewGuid().ToString() +$bpUri = "https://graph.microsoft.com/beta/applications(appId='$BlueprintAppId')" + +Write-Host "Step A: identifierUris -> $appIdUri" +$body = @{ identifierUris = @($appIdUri) } | ConvertTo-Json -Depth 5 +Invoke-MgGraphRequest -Method PATCH -Uri $bpUri -Body $body -ContentType 'application/json' + +Write-Host "Step B: add access_as_user scope (id=$scopeId)" +$scopeBody = @{ + api = @{ + oauth2PermissionScopes = @(@{ + id = $scopeId + adminConsentDescription = 'Access the agent on behalf of the signed-in user' + adminConsentDisplayName = 'Access agent as user' + isEnabled = $true + type = 'User' + userConsentDescription = 'Access the agent on your behalf' + userConsentDisplayName = 'Access agent as user' + value = 'access_as_user' + }) + } +} | ConvertTo-Json -Depth 10 +Invoke-MgGraphRequest -Method PATCH -Uri $bpUri -Body $scopeBody -ContentType 'application/json' + +Write-Host "Step C: verify" +$bp = Invoke-MgGraphRequest -Method GET -Uri ($bpUri + '?$select=identifierUris,api') +$bp | ConvertTo-Json -Depth 10 + +Write-Host "Step D: add requiredResourceAccess on Client SPA" +$spaUri = "https://graph.microsoft.com/v1.0/applications(appId='$ClientSpaAppId')" +$spaBody = @{ + requiredResourceAccess = @(@{ + resourceAppId = $BlueprintAppId + resourceAccess = @(@{ id = $scopeId; type = 'Scope' }) + }) +} | ConvertTo-Json -Depth 10 +Invoke-MgGraphRequest -Method PATCH -Uri $spaUri -Body $spaBody -ContentType 'application/json' + +Write-Host "Step E: admin-consent oauth2PermissionGrant Client SPA -> Blueprint" +$spaSp = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/v1.0/servicePrincipals(appId='$ClientSpaAppId')?`$select=id" +$bpSp = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/v1.0/servicePrincipals(appId='$BlueprintAppId')?`$select=id" +$grantBody = @{ + clientId = $spaSp.id + consentType = 'AllPrincipals' + resourceId = $bpSp.id + scope = 'access_as_user' +} | ConvertTo-Json +try { + Invoke-MgGraphRequest -Method POST -Uri 'https://graph.microsoft.com/v1.0/oauth2PermissionGrants' -Body $grantBody -ContentType 'application/json' | Out-Null + Write-Host " Granted" +} catch { + Write-Host " Note (may already exist): $($_.Exception.Message)" +} + +Write-Host "Done. scopeId=$scopeId" diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh b/.claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh new file mode 100644 index 0000000..6a6721f --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh @@ -0,0 +1,145 @@ +#!/usr/bin/env bash +# smoke-test-kind.sh — validate the AKS manifests on a local `kind` cluster +# without any Azure resources. Uses the ClientSecret credential source +# instead of workload identity (since kind has no Entra-trusted OIDC). +# +# Usage: +# source /tmp/deploy-vars.sh +# export BLUEPRINT_CLIENT_SECRET="" +# bash smoke-test-kind.sh +# bash smoke-test-kind.sh --cleanup +set -euo pipefail + +CLUSTER="agentid-smoke" +NS="agentid" + +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +# Manifests live in this skill's manifests/ directory (sibling of scripts/). +MANIFESTS="$SCRIPT_DIR/../manifests" +# Skill scripts are at .claude/skills/deploy-agent-aks-dev/scripts/ so the workspace +# (or upstream repo) root is 4 up. +REPO_ROOT="$( cd "$SCRIPT_DIR/../../../.." && pwd )" + +# Locate sidecar dev/ and weather-api/ sources. Two supported layouts: +# A) Upstream repo: /sidecar/{dev,weather-api,aks} +# B) Reference clone: /reference/repo/sidecar/{dev,weather-api} +find_sidecar_root() { + local candidate + for candidate in \ + "$REPO_ROOT/sidecar" \ + "$REPO_ROOT/reference/repo/sidecar" ; do + if [[ -d "$candidate/dev" && -d "$candidate/weather-api" ]]; then + ( cd "$candidate" && pwd ); return 0 + fi + done + return 1 +} +SIDECAR_ROOT="$( find_sidecar_root )" || fail "could not locate sidecar/{dev,weather-api}" + +if [[ "${1:-}" == "--cleanup" ]]; then + kind delete cluster --name "$CLUSTER" || true + echo "Cleanup complete." + exit 0 +fi + +fail() { echo "SMOKE FAIL: $1" >&2; exit 1; } +have() { command -v "$1" >/dev/null 2>&1 || fail "missing tool: $1"; } + +have docker +have kind +have kubectl +have envsubst + +for v in TENANT_ID BLUEPRINT_APP_ID AGENT_CLIENT_ID CLIENT_SPA_APP_ID OLLAMA_MODEL; do + [[ -n "${!v:-}" ]] || fail "$v not set (source /tmp/deploy-vars.sh)" +done + +if [[ -z "${BLUEPRINT_CLIENT_SECRET:-}" ]]; then + echo "WARN: BLUEPRINT_CLIENT_SECRET unset — sidecar won't acquire tokens," + echo " but the rest of the wiring will still be validated." + BLUEPRINT_CLIENT_SECRET="placeholder-for-smoke-only" +fi + +echo "==> [1/7] Create kind cluster" +if ! kind get clusters | grep -q "^$CLUSTER$"; then + kind create cluster --name "$CLUSTER" --wait 60s +fi +kubectl cluster-info --context "kind-$CLUSTER" + +echo "==> [2/7] Build images locally" +docker build -t agent-id-dev/llm-agent:smoke "$SIDECAR_ROOT/dev" > /tmp/kind-build-agent.log 2>&1 \ + || { tail -50 /tmp/kind-build-agent.log; fail "image build (llm-agent) — see /tmp/kind-build-agent.log"; } +docker build -t agent-id-dev/weather-api:smoke "$SIDECAR_ROOT/weather-api" > /tmp/kind-build-weather.log 2>&1 \ + || { tail -50 /tmp/kind-build-weather.log; fail "image build (weather-api) — see /tmp/kind-build-weather.log"; } + +echo "==> [3/7] Load images into kind" +kind load docker-image agent-id-dev/llm-agent:smoke --name "$CLUSTER" +kind load docker-image agent-id-dev/weather-api:smoke --name "$CLUSTER" + +echo "==> [4/7] Render manifests (smoke overlay)" +OUT=$(mktemp -d) +ACR_NAME_PROD="${ACR_NAME:-acr-placeholder}" +# Render with envsubst, then rewrite ACR image refs to the locally-loaded tag. +export TENANT_ID BLUEPRINT_APP_ID AGENT_CLIENT_ID CLIENT_SPA_APP_ID OLLAMA_MODEL ACR_NAME="$ACR_NAME_PROD" +for f in "$MANIFESTS"/*.yaml; do + envsubst < "$f" \ + | sed -E "s|${ACR_NAME_PROD}\.azurecr\.io/agent-id-dev/llm-agent:1\.0\.0|agent-id-dev/llm-agent:smoke|g" \ + | sed -E "s|${ACR_NAME_PROD}\.azurecr\.io/agent-id-dev/weather-api:1\.0\.0|agent-id-dev/weather-api:smoke|g" \ + > "$OUT/$(basename "$f")" +done + +# Strip workload-identity bits and inject ClientSecret credential source. +# - Remove "azure.workload.identity/use: true" pod label +# - Replace SignedAssertionFilePath block with ClientSecret block in the sidecar +python3 - "$OUT/40-agent.yaml" "$OUT/10-serviceaccount.yaml" <<'PY' || fail "python3 not found" +import sys, re +agent_f, sa_f = sys.argv[1], sys.argv[2] + +with open(agent_f) as fh: a = fh.read() +a = a.replace('azure.workload.identity/use: "true"', '# (workload-identity disabled in smoke test)') +a = re.sub( + r'- \{ name: AzureAd__ClientCredentials__0__SourceType,[^}]*\}\s*\n' + r'\s*- \{ name: AzureAd__ClientCredentials__0__SignedAssertionFileDiskPath,[^}]*\}', + """- { name: AzureAd__ClientCredentials__0__SourceType, value: "ClientSecret" } + - name: AzureAd__ClientCredentials__0__ClientSecret + valueFrom: + secretKeyRef: { name: blueprint-secret, key: client-secret }""", + a, +) +with open(agent_f, 'w') as fh: fh.write(a) + +with open(sa_f) as fh: s = fh.read() +s = re.sub(r'\n\s+azure\.workload\.identity/[^\n]*', '', s) +with open(sa_f, 'w') as fh: fh.write(s) +PY + +echo "==> [5/7] Apply" +kubectl apply -f "$OUT/00-namespace.yaml" +kubectl -n "$NS" create secret generic blueprint-secret \ + --from-literal=client-secret="$BLUEPRINT_CLIENT_SECRET" \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl apply -f "$OUT/10-serviceaccount.yaml" +kubectl apply -f "$OUT/20-weather-api.yaml" +kubectl apply -f "$OUT/30-ollama.yaml" +kubectl apply -f "$OUT/40-agent.yaml" +# Skip 50-ingress.yaml: kind doesn't have a cloud LB. Use port-forward instead. + +echo "==> [6/7] Wait for rollouts" +kubectl -n "$NS" rollout status deploy/weather-api --timeout=180s || fail "weather-api rollout" +kubectl -n "$NS" rollout status deploy/ollama --timeout=600s || fail "ollama rollout (model pull is slow on first run)" +kubectl -n "$NS" rollout status deploy/llm-agent --timeout=180s || fail "llm-agent rollout" + +echo "==> [7/7] Hit /status" +kubectl -n "$NS" port-forward deploy/llm-agent 3000:3000 >/dev/null 2>&1 & +PF_PID=$! +trap "kill $PF_PID 2>/dev/null || true" EXIT +sleep 3 +STATUS=$(curl -fsS "http://127.0.0.1:3000/status" || true) +echo " /status -> $STATUS" +[[ "$STATUS" == *"ollama_available"* ]] || fail "agent /status did not include ollama_available; got: $STATUS" +[[ "$STATUS" == *'"ollama_available":true'* || "$STATUS" == *"ollama_available: true"* ]] \ + || fail "ollama_available is not true" + +echo +echo "SMOKE PASS — manifests apply, all rollouts succeed, agent reaches Ollama." +echo "Run 'bash $0 --cleanup' to tear down the kind cluster." diff --git a/.claude/skills/teardown-agent-aks-dev/SKILL.md b/.claude/skills/teardown-agent-aks-dev/SKILL.md new file mode 100644 index 0000000..7b75b3a --- /dev/null +++ b/.claude/skills/teardown-agent-aks-dev/SKILL.md @@ -0,0 +1,180 @@ +--- +name: teardown-agent-aks-dev +description: 'AI-led teardown of an Entra Agent ID agent deployed to Azure Kubernetes Service by deploy-agent-aks-dev. Use when an engineering team wants to delete the AKS cluster, the resource group (which removes AKS + ACR + Log Analytics + PVCs in one shot), the Federated Identity Credential added to their Blueprint app, and optionally the Entra apps themselves (Client SPA, Agent Identity, Blueprint). Defaults to DRY-RUN so the operator sees exactly what will be deleted before anything is destroyed. Entra-object deletion is opt-in because Blueprints are often shared. Cross-tenant aware (SUBSCRIPTION_TENANT_ID for the RG delete, TENANT_ID for the FIC delete and Entra object cleanup). NOT for ACA deployments (use teardown-agent-aca-dev), NOT for the AWS variant (use teardown-agent-aca-aws), NOT for local docker-compose stacks (use `docker compose down -v`).' +--- + +# Teardown — Entra Agent ID Agent on AKS (AI-Led) + +Reverses the [`deploy-agent-aks-dev`](../deploy-agent-aks-dev/SKILL.md) skill. Deletes the resource group (AKS, ACR, Log Analytics, PVCs), removes the Federated Identity Credential the deploy added to the Blueprint app, and — opt-in — deletes the Entra apps (Client SPA, Agent Identity, Blueprint). + +**Paired with:** [`deploy-agent-aks-dev`](../deploy-agent-aks-dev/SKILL.md). Uses the same `/tmp/deploy-vars.sh`. + +## When to Use + +- "Tear down the AKS demo", "delete the agent cluster", "clean up the RG". +- Ending a demo / sales engagement and removing billable resources before they accrue ($140+/mo for a small node pool, plus $18/mo for the public LB). +- Re-running the deploy skill from a clean state (e.g. after a misconfigured `NODE_VM_SIZE`). +- Rotating to a different region or subscription — full teardown is cleaner than mutating the cluster. + +## Do NOT Use When + +- **Local docker-compose** — `cd sidecar/dev && docker compose down -v` is enough. +- **ACA variant** — use [`teardown-agent-aca-dev`](../teardown-agent-aca-dev/SKILL.md). +- **AWS variant** — use [`teardown-agent-aca-aws`](../teardown-agent-aca-aws/SKILL.md) (handles IAM role, OIDC provider, intermediary app). +- **Shared / long-lived Blueprint** — confirm with the user that the Blueprint isn't used by other agents before opting into `DELETE_ENTRA=1`. Deleting a shared Blueprint will break every other agent that federates against it. + +## Safety posture + +1. **Dry-run by default** (`DRY_RUN=1`). User must set `DRY_RUN=0` to actually delete anything. +2. **Entra objects are opt-in** (`DELETE_ENTRA=1`). Default keeps the Blueprint, Agent Identity, and Client SPA in the tenant. +3. **Re-confirms before deleting the Blueprint** — even with `DELETE_ENTRA=1`, the orchestrator prompts again, because Blueprints are routinely shared. +4. **FIC delete is automatic** even with `DELETE_ENTRA=0`. The FIC is the only Blueprint-scoped artifact the deploy created, and leaving it behind orphans state without protecting any shared concern. +5. **Confirm tenant + subscription + RG** with the user first. Users frequently have multiple Azure accounts; misfiring an RG delete on the wrong sub is unrecoverable. + +## Prerequisites + +1. `/tmp/deploy-vars.sh` from the original deployment (at minimum `SUBSCRIPTION_ID`, `RG`, `TENANT_ID`, `BLUEPRINT_APP_ID`; `FIC_NAME`, `AGENT_CLIENT_ID`, `CLIENT_SPA_APP_ID` if cleaning those). +2. `az` logged in to **both** tenants if this was a cross-tenant deploy: + ```bash + az login --tenant "${SUBSCRIPTION_TENANT_ID:-$TENANT_ID}" # for RG delete + az login --tenant "$TENANT_ID" # for FIC delete + Entra cleanup + ``` + Single-tenant deploys need only one login. +3. **Graph role**: + - FIC delete needs `Application.ReadWrite.OwnedBy` (own the Blueprint) or `Application.ReadWrite.All`. + - `DELETE_ENTRA=1` additionally needs `Application Administrator` or higher on the apps you're deleting. +4. `pwsh` 7.4+ if running the FIC delete via the `Microsoft.Graph.Authentication` PowerShell path (the orchestrator falls back to `az rest` when `pwsh` isn't available). + +## Procedure + +### Step 0 — Confirm scope with the user + +The orchestrator prints this banner and prompts for confirmation. Reproduce in your message to the user verbatim: + +``` +Teardown plan (AKS / Entra Agent ID): + Subscription: $SUBSCRIPTION_ID + Resource group: $RG (WILL be deleted — removes AKS, ACR, LB, PVCs, logs) + Blueprint app: $BLUEPRINT_APP_ID + └─ FIC to remove: $FIC_NAME (always removed when found) + Delete Entra apps: $DELETE_ENTRA (Client SPA, Agent Identity, Blueprint — opt-in) + Dry run: $DRY_RUN +Proceed? [y/N] +``` + +### Step 1 — Revoke OAuth consent grants on the Agent SP + +Even if you keep the Agent Identity, revoke any user-consent grants so a redeploy starts clean (and so a stale `User.Read` admin-consent isn't left behind on a destroyed cluster). + +```bash +AGENT_SP_OID=$(az ad sp show --id "$AGENT_CLIENT_ID" --query id -o tsv 2>/dev/null || true) +if [[ -n "$AGENT_SP_OID" ]]; then + az rest --method GET \ + --uri "https://graph.microsoft.com/v1.0/oauth2PermissionGrants?\$filter=clientId eq '$AGENT_SP_OID'" \ + --query 'value[].id' -o tsv | while read -r g; do + az rest --method DELETE --uri "https://graph.microsoft.com/v1.0/oauth2PermissionGrants/$g" + done +fi +``` + +### Step 2 — Delete the Federated Identity Credential on the Blueprint + +The deploy added one FIC to the Blueprint (`name = $FIC_NAME`, `subject = system:serviceaccount:agentid:agent-sa`). Remove it so the Blueprint isn't left trusting an OIDC issuer that no longer exists: + +```bash +TENANT_ID="$TENANT_ID" BLUEPRINT_APP_ID="$BLUEPRINT_APP_ID" FIC_NAME="${FIC_NAME:-aks-agent-sa}" \ + bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh --fic-only +``` + +The orchestrator does this automatically in Step 2; the standalone invocation above is for manual triage. + +### Step 3 — Delete the resource group + +```bash +az group delete --name "$RG" --yes --no-wait +``` + +This removes, in one shot: +- The AKS cluster (`$AKS_NAME`) +- The ACR (`$ACR_NAME`) and every image in it +- The system-assigned managed identity AKS provisioned for the kubelet +- Any PVCs (Ollama models) and their backing disks +- The Log Analytics workspace if `ENABLE_LOGS=azure-monitor-container-insights` and it was created in `$RG` +- The Standard LB and its public IP + +> [!NOTE] +> Log Analytics workspaces are sometimes pinned to a different RG by tenant policy. If `az group delete` succeeds but `az monitor log-analytics workspace show` still finds yours, delete it manually. + +### Step 4 — Delete Entra objects (opt-in: `DELETE_ENTRA=1`) + +Asked **per object**, in order, lowest-blast-radius first: + +1. **Client SPA** (`CLIENT_SPA_APP_ID`) — usually safe; created per-deployment. +2. **Agent Identity** — delete via the Agent ID portal or Graph (`DELETE /agentIdentities/{id}`). +3. **Blueprint** (`BLUEPRINT_APP_ID`) — **PROMPT AGAIN.** Often shared across agents. Deleting a shared Blueprint breaks every other agent that federates against it. + +```bash +az ad app delete --id "$CLIENT_SPA_APP_ID" 2>/dev/null || true +# Agent + Blueprint: prompt explicitly first, then call Graph +``` + +### Step 5 — Verify + +```bash +az group exists --name "$RG" # expect: false +az ad app federated-credential list --id "$BLUEPRINT_APP_ID" \ + --query "[?name=='$FIC_NAME']" -o tsv # expect: empty +az ad app show --id "$CLIENT_SPA_APP_ID" 2>&1 | head -1 # expect: "not found" (if DELETE_ENTRA=1) +``` + +## One-Shot Orchestrator + +Single-entry-point script: [`scripts/teardown-aks-dev.sh`](./scripts/teardown-aks-dev.sh). + +```bash +# Dry run (default) — Azure + FIC, no Entra app deletes +bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh + +# Real teardown — RG + FIC, keep Entra apps +DRY_RUN=0 bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh + +# Full teardown — RG + FIC + Entra apps (Client SPA, Agent, Blueprint — each prompted) +DRY_RUN=0 DELETE_ENTRA=1 \ + bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh + +# Just remove the FIC and exit (no RG touch) +bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh --fic-only +``` + +## Cross-tenant teardown + +If the deploy was cross-tenant (Azure sub in tenant A, Entra objects in tenant B), the orchestrator switches `az` context per step: + +| Step | Uses | Tenant | +|---|---|---| +| Revoke OAuth grants | `az rest` against Graph | `$TENANT_ID` (B) | +| Delete FIC | `az rest` / `pwsh + Connect-MgGraph` | `$TENANT_ID` (B) | +| Delete RG | `az group delete` | `$SUBSCRIPTION_TENANT_ID` (A) | +| Delete Entra apps | `az ad app delete` | `$TENANT_ID` (B) | + +You must be signed in to both before running. The orchestrator fails early with a clear message if either context is missing. + +## Common failure modes + +| Symptom | Cause | Fix | +|---|---|---| +| `az group delete` hangs > 10 min | ACR has soft-delete retention enabled, or a KV with purge protection is in the RG | `az resource list -g "$RG"` to find the stuck resource; purge it manually | +| `Authorization_RequestDenied` on `az ad app delete` | Signing-in user lacks `Application Administrator` on the app | Elevate via PIM or have the app owner run it | +| FIC delete returns 404 | Already deleted, or `FIC_NAME` mismatch | `az ad app federated-credential list --id "$BLUEPRINT_APP_ID"` to enumerate actual names | +| `Subscription not found` on `az account set` | Wrong-tenant `az` context (cross-tenant deploy) | `az login --tenant "$SUBSCRIPTION_TENANT_ID"` | +| RG deleted but ACR images still billing | ACR was in a different RG | Find it: `az acr list --query "[?name=='$ACR_NAME']"`; delete: `az acr delete --name "$ACR_NAME" --yes` | +| Baked-in Ollama image still billing after `az group delete` | ACR soft-delete retention window | `az acr list --query "[].{n:name,p:properties.policies.softDeletePolicy.status}"`; purge if present | +| Workload identity webhook errors on next deploy | Old FIC still present with conflicting subject | `--fic-only` mode of the orchestrator, or `az ad app federated-credential delete` by ID | + +## References + +- [Azure — delete resource group](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/delete-resource-group) +- [Microsoft Graph — federatedIdentityCredentials](https://learn.microsoft.com/en-us/graph/api/application-delete-federatedidentitycredentials) +- [Microsoft Graph — oauth2PermissionGrant delete](https://learn.microsoft.com/en-us/graph/api/oauth2permissiongrant-delete) +- [`deploy-agent-aks-dev`](../deploy-agent-aks-dev/SKILL.md) — the deploy skill this reverses +- [`deploy-agent-aks-dev/references/cross-tenant-federation.md`](../deploy-agent-aks-dev/references/cross-tenant-federation.md) — the two-tenant pattern this teardown supports diff --git a/.claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh b/.claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh new file mode 100644 index 0000000..da1e571 --- /dev/null +++ b/.claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh @@ -0,0 +1,234 @@ +#!/usr/bin/env bash +# teardown-aks-dev.sh — orchestrator for teardown-agent-aks-dev. +# +# Safe by default: DRY_RUN=1, DELETE_ENTRA=0. +# +# Usage: +# bash teardown-aks-dev.sh # dry-run, RG + FIC +# DRY_RUN=0 bash teardown-aks-dev.sh # real, RG + FIC +# DRY_RUN=0 DELETE_ENTRA=1 bash teardown-aks-dev.sh # full +# bash teardown-aks-dev.sh --fic-only # just the FIC, no RG touch +# +# Env (from /tmp/deploy-vars.sh): +# SUBSCRIPTION_ID, RG (required for RG mode) +# TENANT_ID, BLUEPRINT_APP_ID (required for FIC delete) +# FIC_NAME (default: aks-agent-sa) +# SUBSCRIPTION_TENANT_ID (cross-tenant; defaults to TENANT_ID) +# AGENT_CLIENT_ID, CLIENT_SPA_APP_ID (optional; needed for DELETE_ENTRA=1) +# +# Exit codes: +# 0 — completed (or dry-run completed) +# 1 — missing required env / preflight failed +# 2 — user aborted at confirmation prompt +# 3 — partial failure (RG deleted but FIC remains, etc.) + +set -u +set -o pipefail + +: "${VARS_FILE:=/tmp/deploy-vars.sh}" +: "${DRY_RUN:=1}" +: "${DELETE_ENTRA:=0}" +: "${FIC_NAME:=aks-agent-sa}" + +FIC_ONLY=0 +for arg in "$@"; do + case "$arg" in + --fic-only) FIC_ONLY=1 ;; + --help|-h) + sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//' + exit 0 + ;; + esac +done + +if [[ -f "$VARS_FILE" ]]; then + # shellcheck disable=SC1090 + source "$VARS_FILE" +else + echo "ERROR: $VARS_FILE not found. Re-export at minimum SUBSCRIPTION_ID, RG, TENANT_ID, BLUEPRINT_APP_ID." >&2 + exit 1 +fi + +: "${SUBSCRIPTION_TENANT_ID:=${TENANT_ID:-}}" +: "${TENANT_ID:?TENANT_ID required in $VARS_FILE}" +: "${BLUEPRINT_APP_ID:?BLUEPRINT_APP_ID required in $VARS_FILE (for FIC delete)}" +if [[ "$FIC_ONLY" -eq 0 ]]; then + : "${SUBSCRIPTION_ID:?SUBSCRIPTION_ID required in $VARS_FILE}" + : "${RG:?RG required in $VARS_FILE}" +fi + +run() { + if [[ "$DRY_RUN" == "1" ]]; then + echo "DRY-RUN: $*" + else + echo "+ $*" + eval "$@" + fi +} + +confirm() { + local prompt="$1" + if [[ "$DRY_RUN" == "1" ]]; then + echo "DRY-RUN: would prompt '$prompt' — assuming yes" + return 0 + fi + read -r -p "$prompt [y/N] " ans + [[ "$ans" == "y" || "$ans" == "Y" ]] +} + +graph_token_for() { + # $1 = tenant id; echoes a Graph access token in that tenant, or empty on failure. + az account get-access-token --tenant "$1" --resource https://graph.microsoft.com \ + --query accessToken -o tsv 2>/dev/null || true +} + +echo "============================================================" +echo "Teardown plan (AKS / Entra Agent ID)" +if [[ "$FIC_ONLY" -eq 0 ]]; then + echo " Subscription tenant: $SUBSCRIPTION_TENANT_ID" + echo " Subscription: $SUBSCRIPTION_ID" + echo " Resource group: $RG (WILL be deleted)" +fi +echo " Entra tenant: $TENANT_ID" +echo " Blueprint app: $BLUEPRINT_APP_ID" +echo " └─ FIC to remove: $FIC_NAME" +if [[ "$FIC_ONLY" -eq 0 ]]; then + echo " Delete Entra apps: $DELETE_ENTRA" +fi +echo " Dry run: $DRY_RUN" +echo " FIC-only mode: $FIC_ONLY" +echo "============================================================" +confirm "Proceed?" || { echo "Aborted."; exit 2; } + +# ---------------------------------------------------------------------- +# FIC-only fast path +# ---------------------------------------------------------------------- +if [[ "$FIC_ONLY" -eq 1 ]]; then + echo "" + echo "Step F — Delete FIC '$FIC_NAME' from Blueprint $BLUEPRINT_APP_ID" + GTOK=$(graph_token_for "$TENANT_ID") + if [[ -z "$GTOK" ]]; then + echo " ERROR: no Graph token for tenant $TENANT_ID. az login --tenant $TENANT_ID first." >&2 + exit 1 + fi + APP_OID=$(curl -sS --fail -H "Authorization: Bearer $GTOK" \ + "https://graph.microsoft.com/v1.0/applications(appId='$BLUEPRINT_APP_ID')?\$select=id" \ + | sed -n 's/.*"id":"\([^"]*\)".*/\1/p') + if [[ -z "$APP_OID" ]]; then + echo " (Blueprint app not found — nothing to do)"; exit 0 + fi + FIC_ID=$(curl -sS --fail -H "Authorization: Bearer $GTOK" \ + "https://graph.microsoft.com/v1.0/applications/$APP_OID/federatedIdentityCredentials" \ + | python3 -c "import json,sys;d=json.load(sys.stdin);print(next((c['id'] for c in d['value'] if c['name']=='$FIC_NAME'),''))") + if [[ -z "$FIC_ID" ]]; then + echo " (FIC '$FIC_NAME' not present — nothing to do)"; exit 0 + fi + run "curl -sS --fail -X DELETE -H 'Authorization: Bearer $GTOK' 'https://graph.microsoft.com/v1.0/applications/$APP_OID/federatedIdentityCredentials/$FIC_ID'" + echo " FIC removed." + exit 0 +fi + +# ---------------------------------------------------------------------- +# Step 1: revoke OAuth grants on Agent SP +# ---------------------------------------------------------------------- +if [[ -n "${AGENT_CLIENT_ID:-}" ]]; then + echo "" + echo "Step 1 — Revoke OAuth consent grants on Agent SP ($AGENT_CLIENT_ID)" + AGENT_SP_OID=$(az ad sp show --id "$AGENT_CLIENT_ID" --query id -o tsv 2>/dev/null || true) + if [[ -n "$AGENT_SP_OID" ]]; then + GRANTS=$(az rest --method GET \ + --uri "https://graph.microsoft.com/v1.0/oauth2PermissionGrants?\$filter=clientId eq '$AGENT_SP_OID'" \ + --query 'value[].id' -o tsv 2>/dev/null || true) + if [[ -n "$GRANTS" ]]; then + while IFS= read -r g; do + [[ -n "$g" ]] && run "az rest --method DELETE --uri 'https://graph.microsoft.com/v1.0/oauth2PermissionGrants/$g'" + done <<< "$GRANTS" + else + echo " (no grants found)" + fi + else + echo " (Agent SP not found — skipping)" + fi +fi + +# ---------------------------------------------------------------------- +# Step 2: delete FIC on Blueprint +# ---------------------------------------------------------------------- +echo "" +echo "Step 2 — Delete FIC '$FIC_NAME' on Blueprint $BLUEPRINT_APP_ID" +GTOK=$(graph_token_for "$TENANT_ID") +if [[ -z "$GTOK" ]]; then + echo " WARN: no Graph token for tenant $TENANT_ID. Skipping FIC delete. (az login --tenant $TENANT_ID and re-run with --fic-only.)" +else + APP_OID=$(curl -sS --fail -H "Authorization: Bearer $GTOK" \ + "https://graph.microsoft.com/v1.0/applications(appId='$BLUEPRINT_APP_ID')?\$select=id" 2>/dev/null \ + | sed -n 's/.*"id":"\([^"]*\)".*/\1/p') + if [[ -n "$APP_OID" ]]; then + FIC_ID=$(curl -sS --fail -H "Authorization: Bearer $GTOK" \ + "https://graph.microsoft.com/v1.0/applications/$APP_OID/federatedIdentityCredentials" \ + | python3 -c "import json,sys;d=json.load(sys.stdin);print(next((c['id'] for c in d['value'] if c['name']=='$FIC_NAME'),''))" 2>/dev/null || true) + if [[ -n "$FIC_ID" ]]; then + run "curl -sS --fail -X DELETE -H 'Authorization: Bearer $GTOK' 'https://graph.microsoft.com/v1.0/applications/$APP_OID/federatedIdentityCredentials/$FIC_ID'" + else + echo " (FIC '$FIC_NAME' not present — skipping)" + fi + else + echo " (Blueprint app not found — skipping)" + fi +fi + +# ---------------------------------------------------------------------- +# Step 3: delete RG +# ---------------------------------------------------------------------- +echo "" +echo "Step 3 — Delete resource group $RG (in sub $SUBSCRIPTION_ID)" +run "az account set --subscription '$SUBSCRIPTION_ID'" +if az group show --name "$RG" >/dev/null 2>&1; then + run "az group delete --name '$RG' --yes --no-wait" +else + echo " (RG does not exist — skipping)" +fi + +# ---------------------------------------------------------------------- +# Step 4: Entra cleanup (opt-in) +# ---------------------------------------------------------------------- +if [[ "$DELETE_ENTRA" == "1" ]]; then + echo "" + echo "Step 4 — Delete Entra objects (in tenant $TENANT_ID)" + if [[ -n "${CLIENT_SPA_APP_ID:-}" ]] && confirm "Delete Client SPA ($CLIENT_SPA_APP_ID)?"; then + run "az ad app delete --id '$CLIENT_SPA_APP_ID' 2>/dev/null || true" + fi + if [[ -n "${AGENT_CLIENT_ID:-}" ]] && confirm "Delete Agent Identity ($AGENT_CLIENT_ID)?"; then + echo " NOTE: delete the Agent Identity via the Agent ID portal or Graph:" + echo " az rest --method DELETE --uri 'https://graph.microsoft.com/beta/agentIdentities/$AGENT_CLIENT_ID'" + fi + echo "" + echo " *** Blueprint ($BLUEPRINT_APP_ID) is often SHARED across agents. ***" + if confirm "Are you SURE you want to delete the Blueprint?"; then + run "az ad app delete --id '$BLUEPRINT_APP_ID' 2>/dev/null || true" + fi +else + echo "" + echo "Step 4 — Skipping Entra cleanup (DELETE_ENTRA=0)" +fi + +# ---------------------------------------------------------------------- +# Step 5: verify +# ---------------------------------------------------------------------- +echo "" +echo "Step 5 — Verify" +if [[ "$DRY_RUN" == "1" ]]; then + echo "DRY-RUN: skipping verification" +else + echo " RG exists? $(az group exists --name "$RG" 2>/dev/null || echo unknown)" + if [[ -n "${BLUEPRINT_APP_ID:-}" ]]; then + REMAIN=$(az ad app federated-credential list --id "$BLUEPRINT_APP_ID" --query "[?name=='$FIC_NAME'] | length(@)" -o tsv 2>/dev/null || echo "?") + echo " FICs named '$FIC_NAME' remaining on Blueprint: $REMAIN" + fi + if [[ "$DELETE_ENTRA" == "1" && -n "${CLIENT_SPA_APP_ID:-}" ]]; then + echo " Client SPA: $(az ad app show --id "$CLIENT_SPA_APP_ID" 2>&1 | head -1)" + fi +fi + +echo "" +echo "Done. If DRY_RUN=1, re-run with DRY_RUN=0 to actually delete." diff --git a/deploy/azure/kubernetes-service/dev/README.md b/deploy/azure/kubernetes-service/dev/README.md new file mode 100644 index 0000000..0272320 --- /dev/null +++ b/deploy/azure/kubernetes-service/dev/README.md @@ -0,0 +1,718 @@ +--- +title: "Tutorial: Deploy a local-LLM agent with the Microsoft Entra Agent ID sidecar on Azure Kubernetes Service" +description: Deploy a self-contained agent (Ollama local LLM + Microsoft Entra Agent ID sidecar) to Azure Kubernetes Service. Workload Identity federation, no stored secrets, one Kubernetes ServiceAccount as the only thing the cluster trusts. +ms.topic: tutorial +ms.date: 05/22/2026 +--- + +# Tutorial: Deploy a local-LLM agent with the Microsoft Entra Agent ID sidecar on Azure Kubernetes Service + +In this tutorial, you deploy a sample AI agent whose model runs **in-cluster on Ollama** and whose identity is brokered by the **Microsoft Entra Agent ID sidecar**. The agent runs as a Kubernetes workload on **Azure Kubernetes Service (AKS)** and authenticates to Microsoft Entra without any long-lived credentials stored in cluster secrets, environment variables, or the container registry. + +The model runs locally inside the cluster, so the deployment has **no second cloud** — the entire token chain begins and ends in Microsoft Entra and the workload boundary is a single Kubernetes pod. That makes this variant the right fit for teams that already standardize on Kubernetes, for organizations with hard requirements on workload portability across clusters (AKS, EKS, GKE, on-prem), and for demos in regulated tenants where the LLM must not leave the customer's network. + +In this tutorial, you learn how to: + +> [!div class="checklist"] +> * Create a Microsoft Entra Agent Identity Blueprint, Agent Identity, and OBO client app. +> * Provision an AKS cluster with the OIDC issuer and Workload Identity webhook enabled, plus a private Azure Container Registry. +> * Federate a Kubernetes ServiceAccount to Microsoft Entra (no managed identity, no client secret, no AWS, no GCP). +> * Build and deploy the agent, sidecar, downstream API, and Ollama as Kubernetes workloads. +> * Verify the autonomous and on-behalf-of (OBO) identity flows end to end. + +> [!TIP] +> **Recommended: AI-assisted deployment.** The fastest, least error-prone way to finish this tutorial is to pair an AI assistant with the skill packaged in this repo: [`.claude/skills/deploy-agent-aks-dev/SKILL.md`](../../../.claude/skills/deploy-agent-aks-dev/SKILL.md). The assistant confirms your SKU choices, picks the right Ollama model strategy, handles the cross-tenant federation case if it applies, and surfaces known failure modes in real time — typically cutting deployment time from hours to minutes. Running the tutorial end-to-end by hand is fully supported (every command is documented below); the skill just front-loads the decisions. +> +> The skill works with **Claude Code** (which reads `.claude/skills/` by default) and with **GitHub Copilot Chat** (ask it to read the `SKILL.md` file). If you prefer a manual run, continue reading — the tutorial remains the source of truth. + +## 1. Overview + +### 1.1 What you build + +A single AKS cluster that exposes a browser UI at `http://`. The cluster runs four workloads under one namespace (`agentid`): + +| Workload | Image | Role | +|---|---|---| +| `llm-agent` Pod (container 1) | `agent-id-dev/llm-agent` (your ACR) | Public-facing Flask + LangChain agent. Receives user chat requests on port **3000**, decides when to call a tool, uses the Ollama HTTP API for LLM completions, and calls `weather-api` for downstream data. | +| `llm-agent` Pod (container 2 — sidecar) | `mcr.microsoft.com/entra-sdk/auth-sidecar` (Microsoft) | The **Microsoft Entra Agent ID auth sidecar**. Listens on `localhost:5000` (pod-internal, not exposed by any Service). `llm-agent` calls it to get Agent Identity tokens — app-only (**TR**, autonomous flow) or on-behalf-of a user (**TU**, OBO flow). Authenticates to Entra as the Blueprint app using `SignedAssertionFilePath` against the projected ServiceAccount token — no client secret. | +| `weather-api` Deployment | `agent-id-dev/weather-api` (your ACR) | Sample downstream API on `8080`, exposed inside the cluster as a `ClusterIP` Service. Validates the Agent Identity JWT on every request (JWKS signature check, issuer, audience, `appid`) and returns real Open-Meteo data only if the call is from the expected Agent Identity. | +| `ollama` Deployment | `ollama/ollama:latest` (public) | Local LLM server on `11434`, exposed inside the cluster as a `ClusterIP` Service. Serves Qwen 2.5 (or another small model) from a `PersistentVolumeClaim` so the model is pulled once and survives pod restarts. | + +**Why one pod for agent + sidecar, and separate Deployments for weather-api and Ollama.** The Entra Agent ID sidecar must share `localhost` with the agent — pod is the smallest Kubernetes unit that guarantees that. The downstream API and the model server are not security-critical co-tenants of the agent; making them independent Deployments lets you scale, restart, swap models, or replace the LLM server (Ollama → vLLM → Azure OpenAI) without touching the agent pod or its auth path. + +**How they're wired:** + +``` +user ──HTTP──▶ Service: llm-agent (LoadBalancer) + │ + ▼ + Pod: llm-agent + │ localhost:5000 ──▶ sidecar (Agent ID tokens) + │ Service:weather-api:8080 ──▶ Pod: weather-api (validates TR/TU) + │ Service:ollama:11434 ──▶ Pod: ollama (local LLM, PVC-backed) + │ + └── (no external model calls — fully self-contained inside the cluster) +``` + +Only the `llm-agent` Service is publicly reachable. The `weather-api` and `ollama` Services are `ClusterIP` and the `sidecar` listener is pod-internal `localhost`, in line with the Entra Agent ID SDK security model. + +When you're done: + +* No `BLUEPRINT_CLIENT_SECRET` exists anywhere in the cluster, in any `Secret`, in any container image, or in the registry. The sidecar authenticates to Entra by reading the projected ServiceAccount token from disk and presenting it as a signed assertion. +* No external model provider credentials exist either — the LLM runs locally inside the cluster. +* Every Entra token rotates automatically (projected ServiceAccount token: 1 hour; Agent Identity tokens: minutes). +* Revocation is a single command: remove the federated credential from the Blueprint app, and the cluster instantly stops being trusted by Entra. + +### 1.2 Architecture + +#### 1.2.1 High-level overview + +``` +┌─────────────────────────────────────────┐ +│ User's browser (MSAL.js) │ +└─────────────────────┬───────────────────┘ + │ HTTP (or HTTPS if you add TLS) + ▼ +┌──────────────────────────────────────────────────────────────────┐ +│ AKS cluster — namespace `agentid` │ +│ │ +│ Service: llm-agent (LoadBalancer) │ +│ │ │ +│ ┌───────────────────▼────────────────────┐ │ +│ │ Pod: llm-agent │ │ +│ │ ├── container: llm-agent │ │ +│ │ └── container: sidecar (Entra SDK) │ │ +│ │ localhost:5000 (pod-internal) │ │ +│ │ reads /var/run/secrets/azure/ │ │ +│ │ tokens/azure-identity-token │ │ +│ └───────────────────┬────────────────────┘ │ +│ │ │ +│ Service: weather-api ──▶ Pod: weather-api │ +│ Service: ollama ──▶ Pod: ollama ──▶ PVC (model cache) │ +│ │ +│ ServiceAccount: agent-sa │ +│ (annotated for Workload Identity) │ +└─────────────────────────────┬────────────────────────────────────┘ + │ + ▼ + Microsoft Entra ID + (Blueprint app — owns the federated credential) +``` + +#### 1.2.2 Identity and federation — one chain, one direction + +There is exactly **one** federation chain in this deployment: the Kubernetes ServiceAccount `agentid/agent-sa` federates to the Blueprint app so the sidecar can sign Entra assertions without a client secret. + +``` + ┌─────────────────────────────────────────┐ + │ Kubernetes ServiceAccount │ + │ agentid/agent-sa │ + │ Issuer = AKS OIDC issuer URL │ + │ Subject = system:serviceaccount: │ + │ agentid:agent-sa │ + └─────────────────────┬───────────────────┘ + │ (sidecar reads the projected token from disk + │ and presents it as a client assertion) + ▼ + ┌─────────────────────────────────────────┐ + │ Blueprint app │ + │ Federated credential: │ + │ issuer = │ + │ subject = system:serviceaccount: │ + │ agentid:agent-sa │ + │ audience = api://AzureADTokenExchange│ + └─────────────────────┬───────────────────┘ + ▼ + Graph + weather-api +``` + +This deployment uses **no user-assigned managed identity, no Azure AD application secret, no AWS OIDC provider, no IAM role**. The ServiceAccount is the only identity the cluster needs to trust, and the federation contract on the Blueprint is the only place Entra needs to be configured. + +### 1.3 Why not static credentials + +A laptop run of this sample uses `BLUEPRINT_CLIENT_SECRET` because it's the simplest pattern for a single developer. On AKS, you promote to `SignedAssertionFilePath`: the **Azure Workload Identity** webhook projects a Kubernetes ServiceAccount token into the pod at `/var/run/secrets/azure/tokens/azure-identity-token`, and the sidecar uses that token as the federated client assertion when calling Entra. The Blueprint client secret disappears from the cluster entirely — there is nothing in any `Secret`, `ConfigMap`, env var, or registry image that can be exfiltrated to obtain a long-lived credential. + +Both patterns are valid — the local one optimizes for simplicity, the cluster one optimizes for secretlessness. They differ only in which value of `AzureAd__ClientCredentials__0__SourceType` the sidecar uses; the agent code is unchanged. + +### 1.4 Why Azure Kubernetes Service + +* **Pod-level sidecar semantics.** Multi-container pods share `localhost` and the same lifecycle, which is exactly what the Entra Agent ID SDK's security model requires. +* **Workload Identity is first-class.** OIDC issuer URL + mutating webhook + projected ServiceAccount tokens are a standard AKS feature flag (`--enable-oidc-issuer --enable-workload-identity`) — no custom controllers, no out-of-band token refresh container. +* **Portable beyond Azure.** The same manifests, with a different OIDC issuer URL, deploy to EKS or GKE. The Entra Agent ID side of the contract is identical because the FIC trusts the issuer URL, not the cloud. +* **Fits existing Kubernetes platforms.** Teams that already operate AKS for other workloads don't need a new compute service. The agent becomes another namespace alongside the rest of the platform. + +## 2. Prerequisites + +### 2.1 Azure + +* A subscription where you can create resource groups, ACR, AKS clusters, public IPs, and (optionally) Log Analytics. +* One of the following Microsoft Entra roles for the signing-in user: + * **Global Administrator**, or + * **Agent ID Administrator** (template ID `db506228-d27e-4b7d-95e5-295956d6615f`), or + * **Agent ID Developer** (template ID `adb2368d-a9be-41b5-8667-d96778e081b0`). +* Application Administrator alone is **not sufficient** — the Blueprint APIs require an Agent ID role. + +> [!NOTE] +> **Cross-tenant supported.** The Blueprint and Agent Identity can live in a different tenant than the Azure subscription that hosts AKS. Workload Identity federation is based on the OIDC issuer URL, not the tenant. See [§7.2](#72-cross-tenant-federation) for the variable layout. + +### 2.2 Tooling + +| Tool | Minimum version | Notes | +|---|---|---| +| `az` CLI | 2.60 | With the `aks-preview` extension: `az extension add --name aks-preview`. | +| `kubectl` | 1.28 | Cluster client. | +| `pwsh` | 7.4 | Required for Agent ID Blueprint Graph operations. | +| `Microsoft.Graph.Authentication` | 2.35 | `Install-Module Microsoft.Graph.Authentication -Scope CurrentUser`. | +| `Microsoft.Graph.Beta.Applications` | 2.35 | Same. | +| `envsubst` | any | Ships with GNU `gettext`; comes with Git Bash on Windows. | +| `kind` *(optional)* | 0.22 | Only needed if you want to run the pre-flight smoke test in [§A](#appendix-a--local-smoke-test-with-kind) without paying for AKS. | +| Docker Desktop *(optional)* | 4.30 | Only required for the `kind` smoke test; AKS builds use `az acr build` and need no local Docker. | + +### 2.3 Repository + +```bash +git clone https://github.com/microsoft/entra-agentid-samples.git +cd entra-agentid-samples +``` + +## 2.5 Choose your SKUs + +Before provisioning anything, pick a SKU for each of the following. The table lists **demo defaults** in bold; the warning blocks describe the silent-failure modes that happen when you accept a default without thinking. All values are set as shell variables in [§4](#4-set-variables). + +| Decision | Variable | Demo default | Alternatives | When to change | +|---|---|---|---|---| +| Node VM size | `NODE_VM_SIZE` | **`Standard_D2s_v5`** (2 vCPU / 8 GiB, ~$70/mo) | `Standard_B2s` (~$30, slower), `Standard_D4s_v5` (~$140), `Standard_D8s_v5` (~$280), GPU `Standard_NC4as_T4_v3` (~$540) | 1.5B model on D2s_v5 CPU is comfortable for `⚡ Direct` calls; LLM-driven tool calling reliability improves materially on D8s_v5 or a GPU pool. | +| Node count | `NODE_COUNT` | **`2`** | `1` (cheaper) … `5` | `1` makes upgrades and replica rescheduling brittle; `≥2` keeps one node free during model pulls. | +| ACR SKU | `ACR_SKU` | **`Basic`** (~$5/mo) | `Standard` (~$20), `Premium` (~$50) | Stay on Basic for demos. Move to Standard if you start pushing multiple model-baked variants. | +| Ollama model | `OLLAMA_MODEL` | **`qwen2.5:1.5b`** (~1.3 GB, CPU-friendly) | `qwen2.5:0.5b`, `qwen2.5:3b`, `qwen2.5:7b` (needs ≥ D8s_v5 or GPU) | Larger models need bigger nodes; reliability and latency of LLM-driven tool calling depend heavily on this. | +| PVC size | `STORAGE_GB` | **`20`** | `10` (1.5B model only), `50` (multiple models cached) | At least `model_disk × 2`. 20 GiB covers any single 7B model with headroom. | +| Ingress | `INGRESS_TYPE` | **`LoadBalancer`** (`Standard` LB, ~$18/mo) | `ingress-nginx` (adds NGINX + cert-manager), `appgw` (AGIC, ~$240/mo) | LoadBalancer is the lightest path to a working demo; switch to `ingress-nginx` or AGIC the moment you need TLS, hostnames, or path-based routing. | +| Logs | `ENABLE_LOGS` | **`none`** (free, kubectl logs only) | `azure-monitor-container-insights` (~$2.76/GB) | Stay on `none` for the first deploy. Enable Container Insights once you hit a "why did this pod crash hours ago" question. | + +> [!WARNING] +> **`NODE_VM_SIZE=Standard_B2s` + `qwen2.5:1.5b`.** The B-series is burstable. Sustained Ollama inference exhausts CPU credits, and answers that should take 1–2 s start taking 30+ s. Use D-series for any demo you'll show live. + +> [!WARNING] +> **`OLLAMA_MODEL=qwen2.5:7b` on D2s_v5.** 7B models on CPU-only 2-vCPU nodes take 15–30 s per turn and often produce hallucinated tool calls under load. Either bump to `D8s_v5`, add a GPU node pool, or accept that LLM-driven tool calling is a stretch goal on small CPU nodes — the **⚡ Direct** mode (which is the authoritative proof of the Entra Agent ID + Workload Identity chain) works on every SKU. + +> [!WARNING] +> **`INGRESS_TYPE=LoadBalancer` + browser sign-in.** Browsers gate `crypto.subtle` on **secure context**, and `http://` is not a secure context. The MSAL.js popup for OBO sign-in throws `pkce_not_created: TypeError: Cannot read properties of undefined (reading 'subtle')`. The workaround is `kubectl port-forward` to `http://localhost:8080` (loopback is exempt). For an end-user-facing demo, switch to `ingress-nginx` with a real TLS certificate. + +> [!WARNING] +> **`ENABLE_LOGS=none` + Ollama init container.** The init container does `ollama pull ` on first replica start (up to 5 min for a 7B model). Without Container Insights you can only inspect this via live `kubectl logs`; once the pod restarts there is no history. Turn logs on for the first deploy. + +For the full decision matrix, see the skill reference: [`sku-sizing.md`](../../../.claude/skills/deploy-agent-aks-dev/references/sku-sizing.md). + +## 3. Final object inventory + +After you finish this tutorial, the following objects exist: + +| Object | Where | Purpose | +|---|---|---| +| Agent Identity Blueprint | Entra | Defines the Agent Identity family. Holds the federated credential that trusts the cluster's ServiceAccount. | +| Agent Identity | Entra | The actual agent principal. Holds Graph app and delegated permissions. | +| Client SPA app | Entra | Browser-side MSAL.js sign-in surface for OBO flows. | +| Resource group | Azure | Container for all Azure resources. | +| Azure Container Registry | Azure | Holds `agent-id-dev/llm-agent` and `agent-id-dev/weather-api`. | +| AKS cluster | Azure | OIDC issuer + Workload Identity webhook enabled; ACR attached for pull. | +| (optional) Log Analytics workspace | Azure | Only if `ENABLE_LOGS=azure-monitor-container-insights`. | +| Kubernetes namespace `agentid` | AKS | Boundary for all in-cluster objects. | +| Kubernetes ServiceAccount `agentid/agent-sa` | AKS | The only thing the cluster needs Entra to trust. | +| Kubernetes Deployments and Services | AKS | `llm-agent`, `weather-api`, `ollama` | +| Kubernetes PersistentVolumeClaim | AKS | Caches the Ollama model across pod restarts. | + +The Blueprint, Agent Identity, and Client SPA app are **tenant-level Entra objects** — they survive cluster deletions. The Azure resource group and the cluster are **disposable**. + +## 4. Set variables + +Run this block once at the start of your shell. Every subsequent command references these variables. The SKU variables come from [§2.5](#25-choose-your-skus) — confirm each choice before you `source` the file. + +```bash +# Azure identity +export TENANT_ID="" # Tenant where the Blueprint & Agent live +export SUBSCRIPTION_TENANT_ID="$TENANT_ID" # Same as TENANT_ID for single-tenant deploy. + # Different value enables cross-tenant deploy (see §7.2). +export SUBSCRIPTION_ID="" +export RG="rg-agentid-aks-dev" +export LOCATION="eastus2" +export AKS_NAME="aks-agentid-dev" +export ACR_NAME="acragentiddev$(openssl rand -hex 3)" # must be globally unique, lowercase, no hyphens + +# SKU decisions (see §2.5 — confirm each one) +export NODE_VM_SIZE="Standard_D2s_v5" +export NODE_COUNT="2" +export ACR_SKU="Basic" +export OLLAMA_MODEL="qwen2.5:1.5b" +export STORAGE_GB="20" +export INGRESS_TYPE="LoadBalancer" # LoadBalancer | ingress-nginx | appgw +export ENABLE_LOGS="none" # none | azure-monitor-container-insights + +# Sign in (two logins for cross-tenant; one for single-tenant) +az login --tenant "$TENANT_ID" # Entra tenant — for Blueprint and FIC operations +az login --tenant "$SUBSCRIPTION_TENANT_ID" # Azure-sub tenant — same as above if single-tenant +az account set --subscription "$SUBSCRIPTION_ID" +``` + +## 5. Phase 1 — Create the Microsoft Entra Agent ID objects + +These Entra objects are independent of the cluster. If you've already created them in a previous tutorial, skip to [§6](#6-phase-2--create-the-azure-infrastructure) and reuse the existing IDs. + +### 5.1 Create the Blueprint and Agent Identity + +```bash +pwsh -NoProfile -Command " +. ./scripts/EntraAgentID-Functions.ps1 +Connect-MgGraph -Scopes ` + 'AgentIdentityBlueprint.AddRemoveCreds.All',` + 'AgentIdentityBlueprint.Create',` + 'AgentIdentityBlueprint.DeleteRestore.All',` + 'AgentIdentity.DeleteRestore.All',` + 'DelegatedPermissionGrant.ReadWrite.All',` + 'Application.Read.All',` + 'AgentIdentityBlueprintPrincipal.Create',` + 'AppRoleAssignment.ReadWrite.All',` + 'Directory.Read.All',` + 'User.Read' -TenantId '$TENANT_ID' -NoWelcome +\$r = Start-EntraAgentIDWorkflow `` + -BlueprintName 'Dev Local-LLM Blueprint' `` + -AgentName 'Local LLM Weather Agent' `` + -Permissions @('User.Read.All') +Write-Host \"BLUEPRINT_APP_ID=\$(\$r.Blueprint.BlueprintAppId)\" +Write-Host \"AGENT_CLIENT_ID=\$(\$r.Agent.AgentIdentityAppId)\" +" +``` + +```bash +export BLUEPRINT_APP_ID="" +export AGENT_CLIENT_ID="" +``` + +### 5.2 Register the Client SPA app + +```bash +cat > scripts/.env < [!div class="checklist"] +> * Blueprint app ID: `$BLUEPRINT_APP_ID` +> * Agent Identity app ID: `$AGENT_CLIENT_ID` +> * Client SPA app ID: `$CLIENT_SPA_APP_ID` + +## 6. Phase 2 — Create the Azure infrastructure + +All commands use the SKU variables set in [§4](#4-set-variables). + +### 6.1 Resource group, ACR, AKS cluster + +```bash +az group create --name "$RG" --location "$LOCATION" -o none + +az acr create --resource-group "$RG" --name "$ACR_NAME" --sku "$ACR_SKU" --admin-enabled false -o none + +# Required resource providers +az provider register --namespace Microsoft.ContainerService --wait +az provider register --namespace Microsoft.ContainerRegistry --wait + +# AKS with OIDC issuer + Workload Identity webhook enabled +CREATE_ARGS=(--resource-group "$RG" --name "$AKS_NAME" --location "$LOCATION" + --node-count "$NODE_COUNT" --node-vm-size "$NODE_VM_SIZE" + --enable-oidc-issuer --enable-workload-identity + --enable-managed-identity + --generate-ssh-keys) + +if [[ "$ENABLE_LOGS" == "azure-monitor-container-insights" ]]; then + az monitor log-analytics workspace create -g "$RG" -n "${AKS_NAME}-logs" -l "$LOCATION" -o none + WS_ID=$(az monitor log-analytics workspace show -g "$RG" -n "${AKS_NAME}-logs" --query id -o tsv) + CREATE_ARGS+=(--enable-addons monitoring --workspace-resource-id "$WS_ID") +fi + +az aks create "${CREATE_ARGS[@]}" -o none +``` + +### 6.2 Attach ACR to AKS + +The kubelet pulls images from ACR using AKS's managed identity. Attaching the registry to the cluster grants the right RBAC automatically. + +```bash +az aks update --resource-group "$RG" --name "$AKS_NAME" --attach-acr "$ACR_NAME" -o none +``` + +### 6.3 Capture the cluster's OIDC issuer URL + +This URL is the trust anchor for the federated credential in [§7](#7-phase-3--federate-the-serviceaccount-to-the-blueprint). + +```bash +export OIDC_ISSUER=$(az aks show -g "$RG" -n "$AKS_NAME" --query oidcIssuerProfile.issuerUrl -o tsv) +echo "$OIDC_ISSUER" +``` + +### 6.4 Get cluster credentials + +```bash +az aks get-credentials --resource-group "$RG" --name "$AKS_NAME" --overwrite-existing +kubectl get nodes +``` + +## 7. Phase 3 — Federate the ServiceAccount to the Blueprint + +The sidecar authenticates to Entra as the Blueprint app using `SignedAssertionFilePath`. Add a federated credential on the Blueprint that trusts the cluster's OIDC issuer and the exact ServiceAccount the agent pod runs under. + +### 7.1 Add the federated credential + +```powershell +pwsh -NoProfile -Command " +Connect-MgGraph -Scopes 'AgentIdentityBlueprint.AddRemoveCreds.All' -TenantId '$env:TENANT_ID' -NoWelcome +\$body = @{ + name = 'aks-agent-sa' + issuer = '$env:OIDC_ISSUER' + subject = 'system:serviceaccount:agentid:agent-sa' + audiences = @('api://AzureADTokenExchange') + description = 'AKS Workload Identity for the local-LLM agent' +} | ConvertTo-Json -Depth 5 +Invoke-MgGraphRequest POST \"https://graph.microsoft.com/beta/applications(appId='$env:BLUEPRINT_APP_ID')/federatedIdentityCredentials\" -Body \$body -ContentType 'application/json' +" +``` + +The sidecar activates this credential by setting `AzureAd__ClientCredentials__0__SourceType=SignedAssertionFilePath` and pointing it at the projected token in [Phase 5](#9-phase-5--deploy-the-kubernetes-workloads). + +**This is the only federation chain in the deployment.** There is no managed identity, no AWS, no GCP, no intermediary app. + +### 7.2 Cross-tenant federation + +Workload Identity federation is **OIDC-URL based**, not tenant-bound — so the Blueprint can live in one tenant while AKS and ACR live in a different tenant's subscription. Common case: an Entra demo tenant holds the Blueprint and Agent Identity, while a corporate billing tenant holds the Azure subscription that runs the cluster. + +Set the two tenant variables independently in [§4](#4-set-variables): + +```bash +export TENANT_ID="" +export SUBSCRIPTION_TENANT_ID="" +``` + +Run `az login` once per tenant; the CLI tracks the two contexts side by side. The federation Graph call always uses `$TENANT_ID` (Blueprint tenant); the cluster commands always use `$SUBSCRIPTION_TENANT_ID` (Azure tenant). Full pattern, variable contract, and common errors: [`cross-tenant-federation.md`](../../../.claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md). + +## 8. Phase 4 — Build and push container images + +Two images go to your ACR: `llm-agent` and `weather-api`. The sidecar image is pulled from MCR at runtime. Ollama uses the upstream image as-is (the model is pulled into the PVC by an `initContainer`, not baked into the image). + +`az acr build` runs the build inside ACR — no local Docker daemon is required. + +```bash +az acr build --registry "$ACR_NAME" \ + --image agent-id-dev/llm-agent:1.0.0 \ + --platform linux/amd64 \ + sidecar/dev + +az acr build --registry "$ACR_NAME" \ + --image agent-id-dev/weather-api:1.0.0 \ + --platform linux/amd64 \ + sidecar/weather-api +``` + +## 9. Phase 5 — Deploy the Kubernetes workloads + +The full manifest set lives in [`.claude/skills/deploy-agent-aks-dev/manifests/`](../../../.claude/skills/deploy-agent-aks-dev/manifests/) and uses `${VAR}` placeholders that `envsubst` substitutes at apply time. + +### 9.1 Render and apply + +```bash +set -a; source /tmp/deploy-vars.sh; set +a # auto-export every variable + +MANIFEST_DIR=".claude/skills/deploy-agent-aks-dev/manifests" + +# Render with explicit varlist so typos fail loudly instead of producing empty strings +VARLIST='$TENANT_ID $BLUEPRINT_APP_ID $AGENT_CLIENT_ID $CLIENT_SPA_APP_ID $ACR_NAME $OLLAMA_MODEL $STORAGE_GB' + +for f in "$MANIFEST_DIR"/*.yaml; do + envsubst "$VARLIST" < "$f" +done | kubectl apply -f - +``` + +The manifests create, in order: + +| File | What it creates | +|---|---| +| `00-namespace.yaml` | Namespace `agentid` | +| `10-serviceaccount.yaml` | ServiceAccount `agent-sa` annotated with `azure.workload.identity/client-id=${BLUEPRINT_APP_ID}` and `tenant-id=${TENANT_ID}` | +| `20-weather-api.yaml` | Deployment + ClusterIP Service for `weather-api` | +| `30-ollama.yaml` | PVC (`${STORAGE_GB}Gi`), `initContainer` that runs `ollama pull ${OLLAMA_MODEL}`, Deployment, ClusterIP Service | +| `40-agent.yaml` | Deployment for the `llm-agent` pod (agent container + sidecar container); pod labelled `azure.workload.identity/use: "true"`; sidecar reads token from `/var/run/secrets/azure/tokens/azure-identity-token` and sets `AzureAd__ClientCredentials__0__SourceType=SignedAssertionFilePath` | +| `50-ingress.yaml` | LoadBalancer Service exposing `llm-agent` on port 80 | + +### 9.2 Wait for the LoadBalancer IP + +```bash +kubectl -n agentid wait deploy/llm-agent --for=condition=available --timeout=10m + +export APP_FQDN=$(kubectl -n agentid get svc llm-agent \ + -o jsonpath='{.status.loadBalancer.ingress[0].ip}') + +echo "http://${APP_FQDN}" +``` + +> [!IMPORTANT] +> The pod label `azure.workload.identity/use: "true"` is what triggers the mutating webhook to project the token into the sidecar's filesystem. If you remove the label, the file `/var/run/secrets/azure/tokens/azure-identity-token` will not exist and the sidecar will log `FileNotFoundException` on every token request. + +## 10. Phase 6 — Post-deployment wiring + +Two manual steps that can't be done before the cluster exists. + +### 10.1 Add the LoadBalancer IP to the Client SPA redirect URIs + +```bash +bash .claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh +``` + +The script PATCHes `spa.redirectUris` on the Client SPA app directly via Graph. It always adds `http://localhost:8080/` (used for the port-forward sign-in path in [§11.4](#114-obo-flow-via-port-forward)) and additionally adds `http://${APP_FQDN}/` if `APP_FQDN` is set. `az ad app update --web-redirect-uris` does **not** affect SPA redirect URIs — that's why this is a Graph PATCH. + +### 10.2 Agent → Graph delegated `User.Read` consent + +Already done in [§5.4](#54-admin-consent-the-agents-delegated-graph-permission). If you skipped it, do it now — you'll hit `AADSTS65001` on the OBO flow otherwise. + +## 11. Phase 7 — Verify + +### 11.1 Confirm pod and Workload Identity wiring + +```bash +kubectl -n agentid get pods +# Expect llm-agent, weather-api, ollama all Running. + +# The sidecar should have the AZURE_* env vars injected by the webhook +kubectl -n agentid exec deploy/llm-agent -c sidecar -- env | grep '^AZURE_' +# Expect: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE, AZURE_AUTHORITY_HOST + +# The projected token file must exist +kubectl -n agentid exec deploy/llm-agent -c sidecar -- \ + ls -l /var/run/secrets/azure/tokens/ +``` + +### 11.2 Status check + +```bash +curl -sS "http://${APP_FQDN}/api/status" | python3 -m json.tool +# Expected: +# "ollama_available": true +# "ollama_model": "qwen2.5:1.5b" +# "sidecar_url": "http://localhost:5000" +``` + +### 11.3 Autonomous flow (⚡ Direct) + +```bash +curl -sS -X POST "http://${APP_FQDN}/api/chat" \ + -H 'Content-Type: application/json' \ + -d '{"message":"Weather in Dallas?","token_flow":"autonomous","use_langchain":false}' \ + | python3 -m json.tool +``` + +The response includes the weather from `weather-api` and Qwen's natural-language explanation. This call exercises the full chain: sidecar → projected token → Entra → Agent Identity token → `weather-api` → Open-Meteo. **This is the authoritative proof that the Entra Agent ID + Workload Identity wiring works.** + +### 11.4 OBO flow (via port-forward) + +Browsers refuse to compute the PKCE challenge over plain HTTP unless the page is loaded from a secure context. `http://` is not a secure context; `http://localhost:*` is. Port-forward to localhost: + +```bash +kubectl -n agentid port-forward svc/llm-agent 8080:80 +``` + +In a browser, open , click **Sign In**, complete MSAL, then chat with **Identity Flow = OBO**. + +For a production-style sign-in URL, terminate TLS in front of the Service (NGINX Ingress + cert-manager, or AGIC + Key Vault). + +### 11.5 Ollama health + +```bash +kubectl -n agentid logs deploy/ollama --tail 30 +kubectl -n agentid exec deploy/ollama -- ollama list +``` + +On first pod start, the `initContainer` runs `ollama pull qwen2.5:1.5b` and writes the model to the PVC. Subsequent restarts skip the pull because the volume persists. + +## 12. Rotate + +Workload Identity rotates its projected ServiceAccount tokens automatically (~1 hour). If you redeploy the cluster, or move tenants: + +1. Capture the new OIDC issuer URL: `az aks show ... --query oidcIssuerProfile.issuerUrl -o tsv`. +2. Delete the Blueprint's federated credential and re-add it with the new `issuer` (and unchanged `subject = system:serviceaccount:agentid:agent-sa`). +3. Update `TENANT_ID` and `BLUEPRINT_APP_ID` annotations on the ServiceAccount if those changed: + ```bash + kubectl -n agentid annotate sa agent-sa azure.workload.identity/client-id=$BLUEPRINT_APP_ID --overwrite + kubectl -n agentid annotate sa agent-sa azure.workload.identity/tenant-id=$TENANT_ID --overwrite + kubectl -n agentid rollout restart deploy/llm-agent + ``` + +There is no AWS or GCP rotation — because there is no AWS or GCP. + +## 13. Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| Sidecar logs `AADSTS70021: No matching federated identity record` | FIC subject mismatch | Recreate the FIC with subject `system:serviceaccount:agentid:agent-sa` — exact characters, no spaces, lowercase. | +| Sidecar logs `AADSTS700016: Application not found` | `AzureAd__ClientId` is the Agent ID, not the Blueprint | Set `AzureAd__ClientId=$BLUEPRINT_APP_ID` in the sidecar env (it's the federated identity holder). | +| `kubectl exec sidecar -- env \| grep AZURE_` returns nothing | Pod missing label `azure.workload.identity/use: "true"` | Add the label to the pod template, redeploy. | +| `AZURE_FEDERATED_TOKEN_FILE` is set but the file doesn't exist | KSA missing the required annotations | Annotate KSA with `azure.workload.identity/client-id=$BLUEPRINT_APP_ID` and `azure.workload.identity/tenant-id=$TENANT_ID`. | +| Sidecar logs `FileNotFoundException: ...azure-identity-token` | Mutating webhook didn't fire (or AKS feature not enabled) | `az aks update -g $RG -n $AKS_NAME --enable-workload-identity`; restart pod. | +| `ollama_available: false` in `/api/status` | initContainer still pulling model | First pull is 1–5 min for `qwen2.5:1.5b`. Tail `kubectl logs deploy/ollama -c init-pull-model`. | +| Ollama pod OOMKilled | Model too large for node | Drop to `qwen2.5:1.5b` or bump `NODE_VM_SIZE`. | +| Agent answers ignore the tool ("here's a generic forecast") | LLM-driven tool calling on small CPU node — small models hallucinate tool decisions | Use **⚡ Direct** mode to verify the auth chain; bump to D8s_v5 + 7B, GPU pool, or Azure OpenAI for reliable Ollama tool calling. | +| OBO sign-in popup throws `pkce_not_created: TypeError: Cannot read properties of undefined (reading 'subtle')` | Browser refuses `crypto.subtle` on non-secure context | Use `kubectl port-forward` and load `http://localhost:8080`. | +| `AADSTS65001` on browser OBO sign-in | Missing delegated `User.Read` admin consent | Run `grant-agent-obo-consent.ps1` (see [§5.4](#54-admin-consent-the-agents-delegated-graph-permission)). | +| `AADSTS50011: redirect URI mismatch` | Deployed URL (or `http://localhost:8080`) not in SPA `redirectUris` | Run `add-spa-redirect-uri.sh` (see [§10.1](#101-add-the-loadbalancer-ip-to-the-client-spa-redirect-uris)). | +| Graph `$filter=appId eq` returns empty for Blueprint | Agent Identity Blueprint types invisible to `$filter` | Use key-lookup form `/beta/applications(appId='')` — the scripts in this skill already do this. | +| `403 Authorization_RequestDenied` on Blueprint create | Signing-in user has only `Application Administrator`, not an Agent ID role | Assign `Agent ID Developer` or `Agent ID Administrator`. | +| Cross-tenant: FIC was added but sidecar still hits `AADSTS70021` | FIC accidentally added to a Blueprint **in the wrong tenant** | Run `Connect-MgGraph -TenantId $TENANT_ID` explicitly before the Graph PATCH. Delete the wrong FIC, recreate in the Blueprint tenant. | +| LB IP stays `` for >5 min | Subscription LB quota exhausted or policy blocks public IPs | Switch `INGRESS_TYPE` to `ingress-nginx` and use an internal LB or an Application Gateway. | +| Pod `ImagePullBackOff` | ACR not attached to AKS, or wrong image name | `az aks update --attach-acr $ACR_NAME`; double-check the manifest image refs match `$ACR_NAME.azurecr.io/agent-id-dev/...:1.0.0`. | +| Rendered manifest still contains `$TENANT_ID` literal | `envsubst` ran without exported vars | `set -a; source /tmp/deploy-vars.sh; set +a` before rendering, or use the explicit varlist form shown in [§9.1](#91-render-and-apply). | + +### 13.1 Diagnostic one-liners + +```bash +# Cluster basics +kubectl -n agentid get pods,svc,sa + +# OIDC issuer (must match the FIC issuer URL on the Blueprint) +az aks show -g "$RG" -n "$AKS_NAME" --query oidcIssuerProfile.issuerUrl -o tsv + +# FIC on the Blueprint +az rest --method GET --url "https://graph.microsoft.com/beta/applications(appId='$BLUEPRINT_APP_ID')/federatedIdentityCredentials" + +# Sidecar env (Workload Identity wiring) +kubectl -n agentid exec deploy/llm-agent -c sidecar -- env | grep '^AZURE_' + +# Sidecar logs (Entra auth errors) +kubectl -n agentid logs deploy/llm-agent -c sidecar --tail 50 + +# Ollama pull progress and served models +kubectl -n agentid logs deploy/ollama --tail 50 +kubectl -n agentid exec deploy/ollama -- ollama list +``` + +## 14. Cost (demo profile, ~24/7) + +| Line item | Approx USD/month | +|---|---| +| AKS — 2 × Standard_D2s_v5 nodes | ~$140 | +| Standard Load Balancer rule + public IP | ~$22 | +| Azure Container Registry Basic | ~$5 | +| `${STORAGE_GB}` GiB managed-csi PVC (default `20`) | ~$1.50 | +| Log Analytics *(only if enabled)* | ~$2 | +| **Total Azure** | **~$170** | +| Per-token model cost | **$0** (Ollama local) | + +`az aks stop --resource-group $RG --name $AKS_NAME` drops the node bill to $0 while preserving the cluster and PVC; total at-rest is ~$30/mo (ACR + LB + PVC + public IP). `az aks start` brings the cluster back in 2–3 min. + +## 15. Clean teardown + +> **TIP — AI-assisted teardown.** If you use Claude Code or GitHub Copilot, invoke the [`teardown-agent-aks-dev`](../../../.claude/skills/teardown-agent-aks-dev/SKILL.md) skill. It runs the same commands below with dry-run by default and prompts at each destructive step. +> +> ```bash +> # Dry run (default — prints commands, deletes nothing) +> bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +> +> # Azure only +> DRY_RUN=0 bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +> +> # Full teardown (Azure + FIC + opt-in Entra apps) +> DRY_RUN=0 DELETE_ENTRA=1 \ +> bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +> ``` + +### 15.1 Order of operations + +1. **Revoke OAuth consent** on the Agent SP so a future redeploy starts from a clean state. +2. **Delete the federated credential** on the Blueprint. The Blueprint itself is usually shared across deployments — do not delete it by accident. +3. **Delete the Azure resource group** — removes the AKS cluster, ACR (with the built images), Log Analytics workspace, public IP, and PVC in one call. +4. **Delete Entra objects** *(opt-in)* — Client SPA, Agent Identity, Blueprint. Blueprints are often shared — **re-confirm before deleting**. + +### 15.2 Manual commands + +```bash +# 0. Load the deployment variables +source /tmp/deploy-vars.sh + +# 1. Revoke OAuth consent on the Agent SP +AGENT_SP_OID=$(az ad sp show --id "$AGENT_CLIENT_ID" --query id -o tsv) +az rest --method GET \ + --uri "https://graph.microsoft.com/v1.0/oauth2PermissionGrants?\$filter=clientId eq '$AGENT_SP_OID'" \ + --query 'value[].id' -o tsv | while read -r g; do + az rest --method DELETE --uri "https://graph.microsoft.com/v1.0/oauth2PermissionGrants/$g" + done + +# 2. Delete the FIC on the Blueprint (does NOT delete the Blueprint app itself) +FIC_ID=$(az rest --method GET \ + --uri "https://graph.microsoft.com/beta/applications(appId='$BLUEPRINT_APP_ID')/federatedIdentityCredentials?\$select=id,name" \ + --query "value[?name=='aks-agent-sa'].id | [0]" -o tsv) +[[ -n "$FIC_ID" ]] && az rest --method DELETE \ + --uri "https://graph.microsoft.com/beta/applications(appId='$BLUEPRINT_APP_ID')/federatedIdentityCredentials/$FIC_ID" + +# 3. Azure — deletes AKS, ACR, public IP, PVC, Log Analytics in one call +az group delete --name "$RG" --yes --no-wait + +# 4. Entra (opt-in) +az ad app delete --id "$CLIENT_SPA_APP_ID" +az rest --method DELETE --uri "https://graph.microsoft.com/beta/agentIdentities/$AGENT_CLIENT_ID" +# Blueprint — re-confirm, this may be shared: +az ad app delete --id "$BLUEPRINT_APP_ID" +``` + +### 15.3 Verify + +```bash +az group exists --name "$RG" # expect: false +az rest --method GET --url "https://graph.microsoft.com/beta/applications(appId='$BLUEPRINT_APP_ID')/federatedIdentityCredentials" \ + --query "value[?name=='aks-agent-sa']" -o tsv # expect: empty +``` + +## Appendix A — Local smoke test with kind + +Before paying for AKS, you can validate the manifest wiring on a local `kind` cluster. The smoke test substitutes `SignedAssertionFilePath` (which needs an Entra-trusted OIDC issuer) with `ClientSecret`, so it does not exercise the federation chain — but it catches typos in the manifests, image build problems, and pod startup issues. + +```bash +source /tmp/deploy-vars.sh +export BLUEPRINT_CLIENT_SECRET="" +bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh + +# Cleanup +bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh --cleanup +``` + +Full details: [`smoke-test.md`](../../../.claude/skills/deploy-agent-aks-dev/references/smoke-test.md). + +## Appendix B — Secretless migration from docker-compose + +| Setting | docker-compose (`sidecar/dev`) | This tutorial (AKS) | +|---|---|---| +| `AzureAd__ClientCredentials__0__SourceType` | `ClientSecret` | `SignedAssertionFilePath` | +| `AzureAd__ClientCredentials__0__SignedAssertionFileDiskPath` | n/a | `/var/run/secrets/azure/tokens/azure-identity-token` | +| `BLUEPRINT_CLIENT_SECRET` | In `.env` | **Deleted** | +| Sidecar network access | Docker bridge | Pod-internal `localhost` | +| FIC on Blueprint | Not required | Required (added in [§7](#7-phase-3--federate-the-serviceaccount-to-the-blueprint)) | +| Identity rotation | Manual secret rotation | Automatic (~1 h projected token rotation) | + +Both configurations are valid: local docker-compose optimizes for setup simplicity; AKS optimizes for secretlessness. From 6d5223cc5d47fe4db30a60d77f136288381183c3 Mon Sep 17 00:00:00 2001 From: vj926 Date: Fri, 22 May 2026 09:52:40 -0700 Subject: [PATCH 2/4] skill(obo): add pre-flight checklist; expand troubleshooting; fix consent grant short-circuit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - references/obo-preflight-checklist.md (new): 12 pre-flight items the agentic CLI walks before enabling OBO — Microsoft.Graph module, admin role, the four GUIDs, SP existence, redirect URI match, secure context, AllPrincipals vs Principal consent decision, assignment gating, clean browser session. Plus 4 post-run verification rows. - references/troubleshooting.md: rows for failure modes seen in real debugging — AADSTS500011 (Blueprint identifierUris empty / SP missing), silent PATCH rollback on platform-managed Blueprint, Connect-MgGraph not recognized (module missing), consentType=Principal trap masking a missing AllPrincipals grant, MSAL browser cache replay, assignment-required gating. - scripts/grant-agent-obo-consent.ps1: fix early-return bug. The pre-existing-grant short-circuit matched any User.Read grant regardless of consentType, so a Principal-typed grant could mask a missing AllPrincipals grant and the script falsely reported success. It now only short-circuits on an AllPrincipals grant and warns when only a Principal grant exists. - SKILL.md: link the new checklist from References. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .claude/skills/deploy-agent-aks-dev/SKILL.md | 1 + .../references/obo-preflight-checklist.md | 56 +++++++++++++++++++ .../references/troubleshooting.md | 21 +++++++ .../scripts/grant-agent-obo-consent.ps1 | 16 +++++- 4 files changed, 91 insertions(+), 3 deletions(-) create mode 100644 .claude/skills/deploy-agent-aks-dev/references/obo-preflight-checklist.md diff --git a/.claude/skills/deploy-agent-aks-dev/SKILL.md b/.claude/skills/deploy-agent-aks-dev/SKILL.md index 8282b48..8a88f65 100644 --- a/.claude/skills/deploy-agent-aks-dev/SKILL.md +++ b/.claude/skills/deploy-agent-aks-dev/SKILL.md @@ -284,6 +284,7 @@ Persisted in `/tmp/deploy-vars.sh`: - [Adapting to EKS / GKE / on-prem](./references/non-azure-k8s.md) — the only Azure-specific pieces and what replaces them - [Local smoke test on `kind`](./references/smoke-test.md) — what's covered, what's not, how to interpret failures - [Troubleshooting matrix](./references/troubleshooting.md) — symptom → cause → fix tables +- [OBO pre-flight checklist](./references/obo-preflight-checklist.md) — **walk this before enabling OBO**; 12 quick checks that catch the failures we hit in real engagements (Microsoft.Graph module, signed-in role, missing SPs, consentType=AllPrincipals vs Principal, browser cache, etc.) ## Paired skills diff --git a/.claude/skills/deploy-agent-aks-dev/references/obo-preflight-checklist.md b/.claude/skills/deploy-agent-aks-dev/references/obo-preflight-checklist.md new file mode 100644 index 0000000..d0258b7 --- /dev/null +++ b/.claude/skills/deploy-agent-aks-dev/references/obo-preflight-checklist.md @@ -0,0 +1,56 @@ +# OBO pre-flight checklist + +Walk through this **before** running any OBO-enablement script +(`setup-obo-blueprint-for-aks.ps1`, `grant-agent-obo-consent.ps1`, +`add-spa-redirect-uri.sh`). Each row is a 30-second check that prevents +a known-painful failure mode we've hit in real customer engagements. + +The agentic CLI **must** verify each item and report status to the user +before proceeding to OBO. If any item is ❌, fix it first. + +| # | Check | How to verify | Why it matters | +|---|---|---|---| +| 1 | `Microsoft.Graph` PowerShell module is installed | `pwsh -Command "Get-Module -ListAvailable Microsoft.Graph.Authentication"` returns a row | All OBO scripts call `Connect-MgGraph`. Missing module → `The term 'Connect-MgGraph' is not recognized`. Install with `Install-Module Microsoft.Graph -Scope CurrentUser -Force`. | +| 2 | Signed in to the **Entra** tenant (not the Azure-sub tenant) | `az account show --query tenantId -o tsv` matches `$TENANT_ID` | Federation lives on the Blueprint app in the Entra tenant. Wrong tenant = silent no-op or `Authorization_RequestDenied`. | +| 3 | Signed-in identity has role to write `oauth2PermissionGrants` and modify app registrations | One of: **Cloud Application Administrator**, **Application Administrator**, **Privileged Role Administrator**, **Global Administrator** | Lower roles let the script run but Entra silently rejects the PATCH/POST. | +| 4 | All four GUIDs known: `TENANT_ID`, `BLUEPRINT_APP_ID`, `AGENT_APP_ID`, `CLIENT_SPA_APP_ID` | Echo each; none should be empty | Wrong/swapped IDs are the #1 cause of `AADSTS500011` later. The OBO audience must be the **Blueprint**, never the Agent. | +| 5 | Blueprint app's Service Principal exists in the tenant | `az ad sp show --id $BLUEPRINT_APP_ID --query id` returns an objectId | `AADSTS500011` says "resource **principal** not found" — App Registration alone isn't enough. If missing: `az ad sp create --id $BLUEPRINT_APP_ID`. | +| 6 | SPA app's Service Principal exists | `az ad sp show --id $CLIENT_SPA_APP_ID --query id` returns an objectId | Required for the `oauth2PermissionGrant` (SPA → Blueprint) and for sign-in. | +| 7 | Agent app's Service Principal exists | `az ad sp show --id $AGENT_APP_ID --query id` returns an objectId | Required for the `oauth2PermissionGrant` (Agent → Graph User.Read). | +| 8 | The agent's actual URL is registered as an SPA redirect URI on the SPA app | `az ad app show --id $CLIENT_SPA_APP_ID --query spa.redirectUris` includes the exact URL the browser will hit (LoadBalancer IP, port-forward `http://localhost:8080`, or HTTPS FQDN) | Mismatch = `AADSTS50011: redirect URI mismatch`. Use `add-spa-redirect-uri.sh` to add. | +| 9 | The browser will hit the SPA over a **secure context** (`https://`) **or** `http://localhost:*` | URL begins with `https://` or `http://localhost` | MSAL.js needs `window.crypto.subtle`. Bare `http://` triggers `pkce_not_created`. Use `scripts/port-forward.sh` for demo. | +| 10 | Consent decision is made up-front: tenant-wide vs per-user | Decide with the user: **AllPrincipals** (everyone in tenant can use, simplest) or **Principal** (per-user grants, brittle) | The default `grant-agent-obo-consent.ps1` ships `AllPrincipals`. If only a subset of users should access the agent, prefer `AllPrincipals` + enable **Assignment required** on the Agent SP (see Row 11). | +| 11 | If only some users should access the agent: `appRoleAssignmentRequired=true` on the Agent SP + users/groups assigned | `az rest --method GET --url "https://graph.microsoft.com/v1.0/servicePrincipals/?\$select=appRoleAssignmentRequired"` returns `true`; `appRoleAssignedTo` lists the intended principals | Without this, tenant-wide consent = anyone in the tenant can use the agent. Assignment-required gates *who* can sign in, independent of consent. | +| 12 | Browser session is clean for testing | Use private/incognito window, or clear site data | MSAL caches the **failed** token request. Without this, a successful fix appears to do nothing because the browser replays the cached failure. | + +## After running the OBO scripts — verification + +These checks confirm the platform actually persisted the writes +(Entra sometimes returns HTTP 204 then silently rolls back changes on +platform-managed Blueprint apps). + +| # | Check | How to verify | If failed | +|---|---|---|---| +| V1 | Blueprint has the OBO Application ID URI | `az ad app show --id $BLUEPRINT_APP_ID --query identifierUris` returns `["api://$BLUEPRINT_APP_ID"]` | PATCH was silently rejected. Try setting it in the Entra portal (Expose an API → Application ID URI). If the portal also refuses or reverts, open a support ticket — Blueprint is platform-locked. See [troubleshooting.md](./troubleshooting.md). | +| V2 | Blueprint exposes `access_as_user` scope | `az ad app show --id $BLUEPRINT_APP_ID --query "api.oauth2PermissionScopes[].value"` includes `access_as_user` | Same as V1 — portal fallback or support ticket. | +| V3 | SPA → Blueprint `access_as_user` grant exists with `consentType=AllPrincipals` | Query `oauth2PermissionGrants` filtered by `clientId=` and `resourceId=` — at least one row with `consentType=AllPrincipals` and `scope` containing `access_as_user` | Re-run with admin role. If a `Principal`-typed grant exists, it does **not** satisfy other users — add an `AllPrincipals` grant. | +| V4 | Agent → Graph `User.Read` grant exists with `consentType=AllPrincipals` | Query `oauth2PermissionGrants` filtered by `clientId=` and `resourceId=` — at least one row with `consentType=AllPrincipals` and `scope` containing `User.Read` | `grant-agent-obo-consent.ps1` may have short-circuited on a pre-existing `Principal` grant. Use the `AllPrincipals` one-liner in [troubleshooting.md](./troubleshooting.md) (`AADSTS65001` row). | + +## One-shot pre-flight script (optional helper) + +If the agent wants a single command to print pass/fail for rows 1–7, +this snippet does it: + +```bash +TENANT_ID="..."; BLUEPRINT_APP_ID="..."; AGENT_APP_ID="..."; CLIENT_SPA_APP_ID="..." + +echo "1) Microsoft.Graph module:"; pwsh -Command "Get-Module -ListAvailable Microsoft.Graph.Authentication" | head -2 +echo "2) Current tenant:"; az account show --query tenantId -o tsv +echo "3) Current identity:"; az ad signed-in-user show --query userPrincipalName -o tsv +echo "4) GUIDs:"; echo " TENANT=$TENANT_ID BP=$BLUEPRINT_APP_ID AGENT=$AGENT_APP_ID SPA=$CLIENT_SPA_APP_ID" +for v in BLUEPRINT_APP_ID AGENT_APP_ID CLIENT_SPA_APP_ID; do + id=$(eval echo \$$v) + printf "5/6/7) SP for %s: " "$v" + az ad sp show --id "$id" --query id -o tsv 2>/dev/null || echo "MISSING — run: az ad sp create --id $id" +done +``` diff --git a/.claude/skills/deploy-agent-aks-dev/references/troubleshooting.md b/.claude/skills/deploy-agent-aks-dev/references/troubleshooting.md index 3d0f900..d3c84e2 100644 --- a/.claude/skills/deploy-agent-aks-dev/references/troubleshooting.md +++ b/.claude/skills/deploy-agent-aks-dev/references/troubleshooting.md @@ -56,10 +56,31 @@ kubectl exec -n agentid deploy/llm-agent -c sidecar -- env | grep ^AZURE_ | Symptom | Cause | Fix | |---|---|---| | OBO sign-in fails with `AADSTS65001` | Agent → Graph User.Read not admin-consented | Run `scripts/grant-agent-obo-consent.ps1` (AKS-local copy) | +| OBO sign-in fails with `AADSTS65001` **after** running `grant-agent-obo-consent.ps1` and it printed "User.Read already granted. Nothing to do." | Existing grant is `consentType=Principal` (per-user, brittle). The script's early-return matched the scope but ignored the consentType. A different signed-in user can't reuse a Principal grant. | Add an `AllPrincipals` grant — use the one-liner below the table. | +| OBO sign-in fails with `AADSTS500011: The resource principal named api:// was not found in the tenant` | Blueprint app's `identifierUris` is empty (or the SP itself doesn't exist) | First confirm SP exists: `az ad sp show --id $BLUEPRINT_APP_ID` — if missing, `az ad sp create --id $BLUEPRINT_APP_ID`. Then run `scripts/setup-obo-blueprint-for-aks.ps1` and confirm the script's verify step prints `✅ identifierUris and access_as_user verified on Blueprint`. | +| `setup-obo-blueprint-for-aks.ps1` exits with `PATCH returned success but Entra did NOT persist the changes` | Blueprint app is platform-managed (`@odata.type: agentIdentityBlueprintPrincipal`, `createdByAppId` = Entra Agent ID first-party SP). Tenant silently rolls back writes to `identifierUris` / scopes. | (1) Re-run as Cloud Application Administrator (or higher). (2) Try setting `Application ID URI` manually in Entra portal → App registrations → Blueprint → Expose an API. (3) If portal also refuses, open a support ticket against the Entra Agent ID team — no client-side fix. | +| `setup-obo-blueprint-for-aks.ps1` or `grant-agent-obo-consent.ps1` errors with `The term 'Connect-MgGraph' is not recognized` | Microsoft Graph PowerShell module not installed on this host | `pwsh -Command "Install-Module Microsoft.Graph -Scope CurrentUser -Force -AllowClobber"`. Minimal install if preferred: `Install-Module Microsoft.Graph.Authentication, Microsoft.Graph.Applications -Scope CurrentUser -Force`. | | OBO sign-in fails with `AADSTS50011: redirect URI mismatch` | Agent FQDN not added to SPA app | Run `scripts/add-spa-redirect-uri.sh` with `APP_FQDN=` | | OBO sign-in works but agent can't call Graph as user | Blueprint not configured for OBO | Re-run `scripts/setup-obo-blueprint-for-aks.ps1` | | weather-api returns 401 to agent | `TENANT_ID` env on weather-api wrong, or `appid` in token doesn't match Agent ID | Confirm both containers see the same `TENANT_ID`; check `kubectl logs deploy/weather-api` for the validation error | | Sign-in popup throws `pkce_not_created: TypeError: Cannot read properties of undefined (reading 'subtle')` | MSAL needs `window.crypto.subtle`, which browsers gate on **secure context**. `http://` is not a secure context; `http://localhost:*` is exempt. | Run `scripts/port-forward.sh` and use `http://localhost:8080` for sign-in. Production-style fix: front the Service with HTTPS (cert-manager + NGINX, or AGIC + Key Vault). | +| OBO fix was applied but browser still shows the old error | MSAL.js caches failed token requests in sessionStorage | Use a **private/incognito** window, or DevTools → Application → Storage → Clear site data, then hard-refresh. | +| OBO works but we only want a subset of users to access the agent | Tenant-wide `AllPrincipals` consent + no assignment gating = anyone in the tenant can sign in | Keep `AllPrincipals` consent; enable **Assignment required** on the Agent SP and assign only the intended users/group. See [obo-preflight-checklist.md](./obo-preflight-checklist.md) row 11. | + +**`AllPrincipals` grant one-liner** (for the second row above — adds tenant-wide Agent → Graph `User.Read`): + +```powershell +pwsh -Command @' +Connect-MgGraph -Scopes "DelegatedPermissionGrant.ReadWrite.All","Application.Read.All" -TenantId "" -NoWelcome +$agent = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/v1.0/servicePrincipals(appId='')?`$select=id" +$graph = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/v1.0/servicePrincipals(appId='00000003-0000-0000-c000-000000000000')?`$select=id" +$body = @{ clientId=$agent.id; consentType="AllPrincipals"; resourceId=$graph.id; scope="User.Read" } | ConvertTo-Json +$r = Invoke-MgGraphRequest -Method POST -Uri "https://graph.microsoft.com/v1.0/oauth2PermissionGrants" -Body $body -ContentType "application/json" +"Granted AllPrincipals. id=$($r.id) scope=$($r.scope)" +'@ +``` + +> **Walk the [OBO pre-flight checklist](./obo-preflight-checklist.md) before enabling OBO** — most of the failures in this table are caught by that 12-row checklist in 2 minutes. ## Cross-tenant federation diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 b/.claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 index aca5b3b..ba9aacf 100644 --- a/.claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 +++ b/.claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 @@ -26,12 +26,22 @@ $existing = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/ Write-Host "Existing grants: $($existing.value.Count)" $existing.value | ForEach-Object { Write-Host " scope='$($_.scope)' consentType=$($_.consentType)" } -$hasUserRead = $existing.value | Where-Object { $_.scope -match 'User\.Read' } | Select-Object -First 1 -if ($hasUserRead) { - Write-Host "User.Read already granted ($($hasUserRead.scope)). Nothing to do." +# IMPORTANT: a Principal-typed (per-user) grant does NOT satisfy other users' +# OBO calls — they'll still hit AADSTS65001. Only short-circuit when a +# tenant-wide AllPrincipals grant already covers User.Read. +$hasAllPrincipalsUserRead = $existing.value | Where-Object { + $_.consentType -eq 'AllPrincipals' -and $_.scope -match '(^|\s)User\.Read(\s|$)' +} | Select-Object -First 1 +if ($hasAllPrincipalsUserRead) { + Write-Host "✅ Tenant-wide (AllPrincipals) User.Read already granted ($($hasAllPrincipalsUserRead.scope)). Nothing to do." return } +$hasPrincipalOnly = $existing.value | Where-Object { $_.consentType -eq 'Principal' -and $_.scope -match 'User\.Read' } | Select-Object -First 1 +if ($hasPrincipalOnly) { + Write-Host "⚠️ Found a Principal (per-user) grant for User.Read — this does NOT cover other users. Adding tenant-wide AllPrincipals grant now..." +} + $body = @{ clientId = $agentSp.id consentType = 'AllPrincipals' From 3e95fdd2f8dacaba2a239611b5bda8786b5cb39b Mon Sep 17 00:00:00 2001 From: vj926 Date: Wed, 10 Jun 2026 09:45:20 -0700 Subject: [PATCH 3/4] Rename AKS skills to deploy-agent-aks-agentid / teardown-agent-aks-agentid Aligns naming with the convention used elsewhere in the repo (-agent--) and disambiguates from the AUID variant landing in PR #33 (deploy-agent-aks-auid). The 'dev' flavor slot is replaced with the explicit identity flavor 'agentid'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../SKILL.md | 28 ++++++++--------- .../manifests/00-namespace.yaml | 0 .../manifests/10-serviceaccount.yaml | 0 .../manifests/20-weather-api.yaml | 0 .../manifests/30-ollama.yaml | 0 .../manifests/40-agent.yaml | 0 .../manifests/50-ingress.yaml | 0 .../references/20-weather-api.yaml | 0 .../references/40-agent.yaml | 0 .../references/architecture.md | 0 .../references/cross-tenant-federation.md | 2 +- .../references/non-azure-k8s.md | 0 .../references/obo-preflight-checklist.md | 0 .../references/post-deploy-manual-steps.md | 6 ++-- .../references/sku-sizing.md | 0 .../references/smoke-test.md | 4 +-- .../references/troubleshooting.md | 0 .../references/workload-identity.md | 0 .../scripts/01-create-aks.sh | 0 .../scripts/02-build-and-push.sh | 0 .../scripts/03-federate-blueprint.ps1 | 0 .../scripts/04-apply-manifests.sh | 0 .../scripts/add-spa-redirect-uri.sh | 0 .../scripts/deploy-aks-dev.sh | 0 .../scripts/deploy-vars.sh.template | 0 .../scripts/grant-agent-obo-consent.ps1 | 0 .../scripts/port-forward.sh | 0 .../scripts/setup-obo-blueprint-for-aks.ps1 | 0 .../scripts/smoke-test-kind.sh | 2 +- .../SKILL.md | 22 +++++++------- .../scripts/teardown-aks-dev.sh | 2 +- deploy/azure/kubernetes-service/dev/README.md | 30 +++++++++---------- 32 files changed, 48 insertions(+), 48 deletions(-) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/SKILL.md (94%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/manifests/00-namespace.yaml (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/manifests/10-serviceaccount.yaml (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/manifests/20-weather-api.yaml (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/manifests/30-ollama.yaml (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/manifests/40-agent.yaml (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/manifests/50-ingress.yaml (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/20-weather-api.yaml (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/40-agent.yaml (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/architecture.md (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/cross-tenant-federation.md (93%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/non-azure-k8s.md (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/obo-preflight-checklist.md (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/post-deploy-manual-steps.md (92%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/sku-sizing.md (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/smoke-test.md (94%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/troubleshooting.md (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/references/workload-identity.md (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/01-create-aks.sh (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/02-build-and-push.sh (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/03-federate-blueprint.ps1 (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/04-apply-manifests.sh (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/add-spa-redirect-uri.sh (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/deploy-aks-dev.sh (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/deploy-vars.sh.template (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/grant-agent-obo-consent.ps1 (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/port-forward.sh (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/setup-obo-blueprint-for-aks.ps1 (100%) rename .claude/skills/{deploy-agent-aks-dev => deploy-agent-aks-agentid}/scripts/smoke-test-kind.sh (98%) rename .claude/skills/{teardown-agent-aks-dev => teardown-agent-aks-agentid}/SKILL.md (81%) rename .claude/skills/{teardown-agent-aks-dev => teardown-agent-aks-agentid}/scripts/teardown-aks-dev.sh (99%) diff --git a/.claude/skills/deploy-agent-aks-dev/SKILL.md b/.claude/skills/deploy-agent-aks-agentid/SKILL.md similarity index 94% rename from .claude/skills/deploy-agent-aks-dev/SKILL.md rename to .claude/skills/deploy-agent-aks-agentid/SKILL.md index 8a88f65..7dd4740 100644 --- a/.claude/skills/deploy-agent-aks-dev/SKILL.md +++ b/.claude/skills/deploy-agent-aks-agentid/SKILL.md @@ -1,6 +1,6 @@ --- -name: deploy-agent-aks-dev -description: 'AI-led, end-to-end deployment of an agent that authenticates with Microsoft Entra Agent ID to Azure Kubernetes Service, using Azure Workload Identity instead of client secrets. Use when an engineering team wants to host their own agent (or this repo''s `sidecar/dev` sample) on AKS with the Entra Agent ID auth-sidecar pattern; when promoting an existing docker-compose stack from ClientSecret to secretless federation; or when an organization already standardized on Kubernetes and needs Agent ID to fit alongside their other workloads. Includes a kind-based local smoke test (no Azure cost), a one-shot orchestrator, a port-forward workflow for the OBO sign-in flow, and an explicit "Adapt for your own agent" section. NOT for Azure Container Apps (use `deploy-agent-aca-dev`), App Service (use `deploy-agent-appservice-dev`), or the AWS Bedrock variant (use `deploy-agent-aca-aws`). Chains to `entra-agent-id-setup` for the Blueprint + Agent Identity + Client SPA objects, and pairs with `teardown-agent-aks-dev` for cleanup.' +name: deploy-agent-aks-agentid +description: 'AI-led, end-to-end deployment of an agent that authenticates with Microsoft Entra Agent ID to Azure Kubernetes Service, using Azure Workload Identity instead of client secrets. Use when an engineering team wants to host their own agent (or this repo''s `sidecar/dev` sample) on AKS with the Entra Agent ID auth-sidecar pattern; when promoting an existing docker-compose stack from ClientSecret to secretless federation; or when an organization already standardized on Kubernetes and needs Agent ID to fit alongside their other workloads. Includes a kind-based local smoke test (no Azure cost), a one-shot orchestrator, a port-forward workflow for the OBO sign-in flow, and an explicit "Adapt for your own agent" section. NOT for Azure Container Apps (use `deploy-agent-aca-dev`), App Service (use `deploy-agent-appservice-dev`), or the AWS Bedrock variant (use `deploy-agent-aca-aws`). Chains to `entra-agent-id-setup` for the Blueprint + Agent Identity + Client SPA objects, and pairs with `teardown-agent-aks-agentid` for cleanup.' --- # Deploy an Entra Agent ID Agent to Azure Kubernetes Service (AI-Led) @@ -72,7 +72,7 @@ The procedure is built for the included `sidecar/dev` sample. If you're bringing ### Step 0 — Confirm account and populate variables ```bash -cp .claude/skills/deploy-agent-aks-dev/scripts/deploy-vars.sh.template /tmp/deploy-vars.sh +cp .claude/skills/deploy-agent-aks-agentid/scripts/deploy-vars.sh.template /tmp/deploy-vars.sh # Edit /tmp/deploy-vars.sh: fill in TENANT_ID, SUBSCRIPTION_ID, RG, LOCATION, SKUs. source /tmp/deploy-vars.sh @@ -90,7 +90,7 @@ Delegate to [`entra-agent-id-setup`](../entra-agent-id-setup/SKILL.md). Capture Then configure the Blueprint for OBO (sets `identifierUris`, adds the `access_as_user` scope, pre-authorizes the Client SPA, and pre-grants admin consent — all idempotent): ```bash -pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 \ +pwsh -NoProfile -File .claude/skills/deploy-agent-aks-agentid/scripts/setup-obo-blueprint-for-aks.ps1 \ -BlueprintAppId "$BLUEPRINT_APP_ID" \ -ClientSpaAppId "$CLIENT_SPA_APP_ID" \ -AgentAppId "$AGENT_CLIENT_ID" \ @@ -104,7 +104,7 @@ Skip this script if the deployment is autonomous-only (no user sign-in). It's sa Validate every manifest against a real Kubernetes API server with no Azure cost. The smoke test uses `ClientSecret` for the sidecar (matches the upstream docker-compose), so no federation is required. ```bash -bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh ``` What it covers and what it doesn't: [references/smoke-test.md](./references/smoke-test.md). Output is one line — `SMOKE PASS` or `SMOKE FAIL: `. **Do this before Step 2** unless you're already comfortable with the manifests. @@ -112,7 +112,7 @@ What it covers and what it doesn't: [references/smoke-test.md](./references/smok ### Step 2 — Azure infrastructure (RG + ACR + AKS with OIDC + Workload Identity) ```bash -bash .claude/skills/deploy-agent-aks-dev/scripts/01-create-aks.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/01-create-aks.sh ``` Creates: @@ -128,7 +128,7 @@ Creates: ### Step 3 — Federate the KSA to the Blueprint app (the only federation chain) ```bash -pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/03-federate-blueprint.ps1 \ +pwsh -NoProfile -File .claude/skills/deploy-agent-aks-agentid/scripts/03-federate-blueprint.ps1 \ -TenantId "$TENANT_ID" \ -BlueprintAppId "$BLUEPRINT_APP_ID" \ -OidcIssuerUrl "$OIDC_ISSUER" \ @@ -145,7 +145,7 @@ Adds one Federated Identity Credential on the Blueprint app: ### Step 4 — Build and push container images ```bash -bash .claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/02-build-and-push.sh ``` `az acr build` for `llm-agent` and `weather-api`. **No local Docker required.** Ollama uses the upstream `ollama/ollama:latest` image as-is — the model is fetched by an initContainer on first pod start and persisted in a PVC. If your tenant has an Azure Policy blocking public Docker Hub pulls, pre-import: `az acr import --name "$ACR_NAME" --source docker.io/ollama/ollama:latest` and update `30-ollama.yaml` to reference the ACR copy. @@ -153,7 +153,7 @@ bash .claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh ### Step 5 — Apply manifests ```bash -bash .claude/skills/deploy-agent-aks-dev/scripts/04-apply-manifests.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/04-apply-manifests.sh ``` Renders `manifests/*.yaml` through `envsubst` (with an **explicit varlist** — `$TENANT_ID $BLUEPRINT_APP_ID $AGENT_CLIENT_ID $ACR_NAME $OLLAMA_MODEL $CLIENT_SPA_APP_ID` — to avoid clobbering shell variables like `$PID` inside init scripts), then `kubectl apply -f -`, then `kubectl rollout status` per Deployment, and finally waits for the LoadBalancer external IP. Captures `APP_FQDN=` into `/tmp/deploy-vars.sh`. @@ -167,20 +167,20 @@ Renders `manifests/*.yaml` through `envsubst` (with an **explicit varlist** — ```bash APP_FQDN="$APP_FQDN" \ - bash .claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh + bash .claude/skills/deploy-agent-aks-agentid/scripts/add-spa-redirect-uri.sh ``` 2. **Grant Agent → Graph delegated `User.Read`** (fixes `AADSTS65001` on OBO): ```bash - pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 \ + pwsh -NoProfile -File .claude/skills/deploy-agent-aks-agentid/scripts/grant-agent-obo-consent.ps1 \ -AgentAppId "$AGENT_CLIENT_ID" -TenantId "$TENANT_ID" ``` 3. **Use port-forward for OBO sign-in.** The LoadBalancer is plain HTTP, which browsers refuse to treat as a "secure context" — MSAL's PKCE flow needs `crypto.subtle`, which is gated on secure-context, so the sign-in popup never opens on `http://`. Loopback is exempt: ```bash - bash .claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh + bash .claude/skills/deploy-agent-aks-agentid/scripts/port-forward.sh # browser: http://localhost:8080 → click "Sign In" ``` @@ -250,7 +250,7 @@ When prereqs are met and SKU variables confirmed: ```bash source /tmp/deploy-vars.sh -bash .claude/skills/deploy-agent-aks-dev/scripts/deploy-aks-dev.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/deploy-aks-dev.sh ``` Idempotent. Runs Steps 2 → 5 in order. Steps 0, 1, A, 6 require human decisions or interactive sign-in and remain manual. @@ -289,5 +289,5 @@ Persisted in `/tmp/deploy-vars.sh`: ## Paired skills - **Setup of Entra objects:** [`entra-agent-id-setup`](../entra-agent-id-setup/SKILL.md) — creates Blueprint + Agent Identity + Client SPA. -- **Teardown:** [`teardown-agent-aks-dev`](../teardown-agent-aks-dev/SKILL.md) — reverses this skill. DRY-RUN by default. Cleans the RG, the FIC on the Blueprint, and (opt-in) the Entra apps. +- **Teardown:** [`teardown-agent-aks-agentid`](../teardown-agent-aks-agentid/SKILL.md) — reverses this skill. DRY-RUN by default. Cleans the RG, the FIC on the Blueprint, and (opt-in) the Entra apps. - **Alternate hosting:** [`deploy-agent-aca-dev`](../deploy-agent-aca-dev/SKILL.md) — same agent, Azure Container Apps instead of AKS. Use when the team is not already on Kubernetes. diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/00-namespace.yaml b/.claude/skills/deploy-agent-aks-agentid/manifests/00-namespace.yaml similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/manifests/00-namespace.yaml rename to .claude/skills/deploy-agent-aks-agentid/manifests/00-namespace.yaml diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/10-serviceaccount.yaml b/.claude/skills/deploy-agent-aks-agentid/manifests/10-serviceaccount.yaml similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/manifests/10-serviceaccount.yaml rename to .claude/skills/deploy-agent-aks-agentid/manifests/10-serviceaccount.yaml diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/20-weather-api.yaml b/.claude/skills/deploy-agent-aks-agentid/manifests/20-weather-api.yaml similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/manifests/20-weather-api.yaml rename to .claude/skills/deploy-agent-aks-agentid/manifests/20-weather-api.yaml diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/30-ollama.yaml b/.claude/skills/deploy-agent-aks-agentid/manifests/30-ollama.yaml similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/manifests/30-ollama.yaml rename to .claude/skills/deploy-agent-aks-agentid/manifests/30-ollama.yaml diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/40-agent.yaml b/.claude/skills/deploy-agent-aks-agentid/manifests/40-agent.yaml similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/manifests/40-agent.yaml rename to .claude/skills/deploy-agent-aks-agentid/manifests/40-agent.yaml diff --git a/.claude/skills/deploy-agent-aks-dev/manifests/50-ingress.yaml b/.claude/skills/deploy-agent-aks-agentid/manifests/50-ingress.yaml similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/manifests/50-ingress.yaml rename to .claude/skills/deploy-agent-aks-agentid/manifests/50-ingress.yaml diff --git a/.claude/skills/deploy-agent-aks-dev/references/20-weather-api.yaml b/.claude/skills/deploy-agent-aks-agentid/references/20-weather-api.yaml similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/references/20-weather-api.yaml rename to .claude/skills/deploy-agent-aks-agentid/references/20-weather-api.yaml diff --git a/.claude/skills/deploy-agent-aks-dev/references/40-agent.yaml b/.claude/skills/deploy-agent-aks-agentid/references/40-agent.yaml similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/references/40-agent.yaml rename to .claude/skills/deploy-agent-aks-agentid/references/40-agent.yaml diff --git a/.claude/skills/deploy-agent-aks-dev/references/architecture.md b/.claude/skills/deploy-agent-aks-agentid/references/architecture.md similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/references/architecture.md rename to .claude/skills/deploy-agent-aks-agentid/references/architecture.md diff --git a/.claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md b/.claude/skills/deploy-agent-aks-agentid/references/cross-tenant-federation.md similarity index 93% rename from .claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md rename to .claude/skills/deploy-agent-aks-agentid/references/cross-tenant-federation.md index 04e4c6d..dee0a03 100644 --- a/.claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md +++ b/.claude/skills/deploy-agent-aks-agentid/references/cross-tenant-federation.md @@ -61,4 +61,4 @@ For `add-spa-redirect-uri.sh` the script calls `az account get-access-token --te ## Teardown caveat -The companion `teardown-agent-aks-dev` skill uses the same split: `SUBSCRIPTION_TENANT_ID` for the RG delete, `TENANT_ID` for FIC delete on the Blueprint and (opt-in) Entra-object deletes. Keep both vars in `/tmp/deploy-vars.sh` so teardown can target them correctly. +The companion `teardown-agent-aks-agentid` skill uses the same split: `SUBSCRIPTION_TENANT_ID` for the RG delete, `TENANT_ID` for FIC delete on the Blueprint and (opt-in) Entra-object deletes. Keep both vars in `/tmp/deploy-vars.sh` so teardown can target them correctly. diff --git a/.claude/skills/deploy-agent-aks-dev/references/non-azure-k8s.md b/.claude/skills/deploy-agent-aks-agentid/references/non-azure-k8s.md similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/references/non-azure-k8s.md rename to .claude/skills/deploy-agent-aks-agentid/references/non-azure-k8s.md diff --git a/.claude/skills/deploy-agent-aks-dev/references/obo-preflight-checklist.md b/.claude/skills/deploy-agent-aks-agentid/references/obo-preflight-checklist.md similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/references/obo-preflight-checklist.md rename to .claude/skills/deploy-agent-aks-agentid/references/obo-preflight-checklist.md diff --git a/.claude/skills/deploy-agent-aks-dev/references/post-deploy-manual-steps.md b/.claude/skills/deploy-agent-aks-agentid/references/post-deploy-manual-steps.md similarity index 92% rename from .claude/skills/deploy-agent-aks-dev/references/post-deploy-manual-steps.md rename to .claude/skills/deploy-agent-aks-agentid/references/post-deploy-manual-steps.md index 8d6e4aa..5e05c71 100644 --- a/.claude/skills/deploy-agent-aks-dev/references/post-deploy-manual-steps.md +++ b/.claude/skills/deploy-agent-aks-agentid/references/post-deploy-manual-steps.md @@ -15,7 +15,7 @@ The Client SPA app was registered with only `http://localhost:3003`. For browser ```bash APP_FQDN="$APP_FQDN" \ - bash .claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh + bash .claude/skills/deploy-agent-aks-agentid/scripts/add-spa-redirect-uri.sh ``` The script idempotently appends both `http://localhost:8080/` and `http://$APP_FQDN/` to `spa.redirectUris`. @@ -39,7 +39,7 @@ AADSTS65001: The user or administrator has not consented to use the application ### Fix ```powershell -pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 ` +pwsh -NoProfile -File .claude/skills/deploy-agent-aks-agentid/scripts/grant-agent-obo-consent.ps1 ` -AgentAppId "$env:AGENT_CLIENT_ID" -TenantId "$env:TENANT_ID" ``` @@ -48,7 +48,7 @@ Idempotent — checks for an existing grant first. Creates `oauth2PermissionGran ## 3. Open the agent via port-forward to exercise OBO ```bash -bash .claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/port-forward.sh # in another shell / browser: # http://localhost:8080 ``` diff --git a/.claude/skills/deploy-agent-aks-dev/references/sku-sizing.md b/.claude/skills/deploy-agent-aks-agentid/references/sku-sizing.md similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/references/sku-sizing.md rename to .claude/skills/deploy-agent-aks-agentid/references/sku-sizing.md diff --git a/.claude/skills/deploy-agent-aks-dev/references/smoke-test.md b/.claude/skills/deploy-agent-aks-agentid/references/smoke-test.md similarity index 94% rename from .claude/skills/deploy-agent-aks-dev/references/smoke-test.md rename to .claude/skills/deploy-agent-aks-agentid/references/smoke-test.md index ba5fd7c..f71493f 100644 --- a/.claude/skills/deploy-agent-aks-dev/references/smoke-test.md +++ b/.claude/skills/deploy-agent-aks-agentid/references/smoke-test.md @@ -30,12 +30,12 @@ So the smoke test proves the **kubernetes wiring** is correct. The **secretless source /tmp/deploy-vars.sh # for TENANT_ID, BLUEPRINT_APP_ID, *_CLIENT_ID export BLUEPRINT_CLIENT_SECRET="" -bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh # ... runs for 5-10 min on first run (mostly Ollama model pull) ... # Last line: SMOKE PASS or SMOKE FAIL: # Cleanup: -bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh --cleanup +bash .claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh --cleanup ``` ## How it differs from production manifests diff --git a/.claude/skills/deploy-agent-aks-dev/references/troubleshooting.md b/.claude/skills/deploy-agent-aks-agentid/references/troubleshooting.md similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/references/troubleshooting.md rename to .claude/skills/deploy-agent-aks-agentid/references/troubleshooting.md diff --git a/.claude/skills/deploy-agent-aks-dev/references/workload-identity.md b/.claude/skills/deploy-agent-aks-agentid/references/workload-identity.md similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/references/workload-identity.md rename to .claude/skills/deploy-agent-aks-agentid/references/workload-identity.md diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/01-create-aks.sh b/.claude/skills/deploy-agent-aks-agentid/scripts/01-create-aks.sh similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/01-create-aks.sh rename to .claude/skills/deploy-agent-aks-agentid/scripts/01-create-aks.sh diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh b/.claude/skills/deploy-agent-aks-agentid/scripts/02-build-and-push.sh similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/02-build-and-push.sh rename to .claude/skills/deploy-agent-aks-agentid/scripts/02-build-and-push.sh diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/03-federate-blueprint.ps1 b/.claude/skills/deploy-agent-aks-agentid/scripts/03-federate-blueprint.ps1 similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/03-federate-blueprint.ps1 rename to .claude/skills/deploy-agent-aks-agentid/scripts/03-federate-blueprint.ps1 diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/04-apply-manifests.sh b/.claude/skills/deploy-agent-aks-agentid/scripts/04-apply-manifests.sh similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/04-apply-manifests.sh rename to .claude/skills/deploy-agent-aks-agentid/scripts/04-apply-manifests.sh diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh b/.claude/skills/deploy-agent-aks-agentid/scripts/add-spa-redirect-uri.sh similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh rename to .claude/skills/deploy-agent-aks-agentid/scripts/add-spa-redirect-uri.sh diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/deploy-aks-dev.sh b/.claude/skills/deploy-agent-aks-agentid/scripts/deploy-aks-dev.sh similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/deploy-aks-dev.sh rename to .claude/skills/deploy-agent-aks-agentid/scripts/deploy-aks-dev.sh diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/deploy-vars.sh.template b/.claude/skills/deploy-agent-aks-agentid/scripts/deploy-vars.sh.template similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/deploy-vars.sh.template rename to .claude/skills/deploy-agent-aks-agentid/scripts/deploy-vars.sh.template diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 b/.claude/skills/deploy-agent-aks-agentid/scripts/grant-agent-obo-consent.ps1 similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 rename to .claude/skills/deploy-agent-aks-agentid/scripts/grant-agent-obo-consent.ps1 diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh b/.claude/skills/deploy-agent-aks-agentid/scripts/port-forward.sh similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/port-forward.sh rename to .claude/skills/deploy-agent-aks-agentid/scripts/port-forward.sh diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 b/.claude/skills/deploy-agent-aks-agentid/scripts/setup-obo-blueprint-for-aks.ps1 similarity index 100% rename from .claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 rename to .claude/skills/deploy-agent-aks-agentid/scripts/setup-obo-blueprint-for-aks.ps1 diff --git a/.claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh b/.claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh similarity index 98% rename from .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh rename to .claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh index 6a6721f..9a62c2e 100644 --- a/.claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh +++ b/.claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh @@ -16,7 +16,7 @@ NS="agentid" SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" # Manifests live in this skill's manifests/ directory (sibling of scripts/). MANIFESTS="$SCRIPT_DIR/../manifests" -# Skill scripts are at .claude/skills/deploy-agent-aks-dev/scripts/ so the workspace +# Skill scripts are at .claude/skills/deploy-agent-aks-agentid/scripts/ so the workspace # (or upstream repo) root is 4 up. REPO_ROOT="$( cd "$SCRIPT_DIR/../../../.." && pwd )" diff --git a/.claude/skills/teardown-agent-aks-dev/SKILL.md b/.claude/skills/teardown-agent-aks-agentid/SKILL.md similarity index 81% rename from .claude/skills/teardown-agent-aks-dev/SKILL.md rename to .claude/skills/teardown-agent-aks-agentid/SKILL.md index 7b75b3a..a91288a 100644 --- a/.claude/skills/teardown-agent-aks-dev/SKILL.md +++ b/.claude/skills/teardown-agent-aks-agentid/SKILL.md @@ -1,13 +1,13 @@ --- -name: teardown-agent-aks-dev -description: 'AI-led teardown of an Entra Agent ID agent deployed to Azure Kubernetes Service by deploy-agent-aks-dev. Use when an engineering team wants to delete the AKS cluster, the resource group (which removes AKS + ACR + Log Analytics + PVCs in one shot), the Federated Identity Credential added to their Blueprint app, and optionally the Entra apps themselves (Client SPA, Agent Identity, Blueprint). Defaults to DRY-RUN so the operator sees exactly what will be deleted before anything is destroyed. Entra-object deletion is opt-in because Blueprints are often shared. Cross-tenant aware (SUBSCRIPTION_TENANT_ID for the RG delete, TENANT_ID for the FIC delete and Entra object cleanup). NOT for ACA deployments (use teardown-agent-aca-dev), NOT for the AWS variant (use teardown-agent-aca-aws), NOT for local docker-compose stacks (use `docker compose down -v`).' +name: teardown-agent-aks-agentid +description: 'AI-led teardown of an Entra Agent ID agent deployed to Azure Kubernetes Service by deploy-agent-aks-agentid. Use when an engineering team wants to delete the AKS cluster, the resource group (which removes AKS + ACR + Log Analytics + PVCs in one shot), the Federated Identity Credential added to their Blueprint app, and optionally the Entra apps themselves (Client SPA, Agent Identity, Blueprint). Defaults to DRY-RUN so the operator sees exactly what will be deleted before anything is destroyed. Entra-object deletion is opt-in because Blueprints are often shared. Cross-tenant aware (SUBSCRIPTION_TENANT_ID for the RG delete, TENANT_ID for the FIC delete and Entra object cleanup). NOT for ACA deployments (use teardown-agent-aca-dev), NOT for the AWS variant (use teardown-agent-aca-aws), NOT for local docker-compose stacks (use `docker compose down -v`).' --- # Teardown — Entra Agent ID Agent on AKS (AI-Led) -Reverses the [`deploy-agent-aks-dev`](../deploy-agent-aks-dev/SKILL.md) skill. Deletes the resource group (AKS, ACR, Log Analytics, PVCs), removes the Federated Identity Credential the deploy added to the Blueprint app, and — opt-in — deletes the Entra apps (Client SPA, Agent Identity, Blueprint). +Reverses the [`deploy-agent-aks-agentid`](../deploy-agent-aks-agentid/SKILL.md) skill. Deletes the resource group (AKS, ACR, Log Analytics, PVCs), removes the Federated Identity Credential the deploy added to the Blueprint app, and — opt-in — deletes the Entra apps (Client SPA, Agent Identity, Blueprint). -**Paired with:** [`deploy-agent-aks-dev`](../deploy-agent-aks-dev/SKILL.md). Uses the same `/tmp/deploy-vars.sh`. +**Paired with:** [`deploy-agent-aks-agentid`](../deploy-agent-aks-agentid/SKILL.md). Uses the same `/tmp/deploy-vars.sh`. ## When to Use @@ -83,7 +83,7 @@ The deploy added one FIC to the Blueprint (`name = $FIC_NAME`, `subject = system ```bash TENANT_ID="$TENANT_ID" BLUEPRINT_APP_ID="$BLUEPRINT_APP_ID" FIC_NAME="${FIC_NAME:-aks-agent-sa}" \ - bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh --fic-only + bash .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh --fic-only ``` The orchestrator does this automatically in Step 2; the standalone invocation above is for manual triage. @@ -133,17 +133,17 @@ Single-entry-point script: [`scripts/teardown-aks-dev.sh`](./scripts/teardown-ak ```bash # Dry run (default) — Azure + FIC, no Entra app deletes -bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +bash .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh # Real teardown — RG + FIC, keep Entra apps -DRY_RUN=0 bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +DRY_RUN=0 bash .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh # Full teardown — RG + FIC + Entra apps (Client SPA, Agent, Blueprint — each prompted) DRY_RUN=0 DELETE_ENTRA=1 \ - bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh + bash .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh # Just remove the FIC and exit (no RG touch) -bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh --fic-only +bash .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh --fic-only ``` ## Cross-tenant teardown @@ -176,5 +176,5 @@ You must be signed in to both before running. The orchestrator fails early with - [Azure — delete resource group](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/delete-resource-group) - [Microsoft Graph — federatedIdentityCredentials](https://learn.microsoft.com/en-us/graph/api/application-delete-federatedidentitycredentials) - [Microsoft Graph — oauth2PermissionGrant delete](https://learn.microsoft.com/en-us/graph/api/oauth2permissiongrant-delete) -- [`deploy-agent-aks-dev`](../deploy-agent-aks-dev/SKILL.md) — the deploy skill this reverses -- [`deploy-agent-aks-dev/references/cross-tenant-federation.md`](../deploy-agent-aks-dev/references/cross-tenant-federation.md) — the two-tenant pattern this teardown supports +- [`deploy-agent-aks-agentid`](../deploy-agent-aks-agentid/SKILL.md) — the deploy skill this reverses +- [`deploy-agent-aks-agentid/references/cross-tenant-federation.md`](../deploy-agent-aks-agentid/references/cross-tenant-federation.md) — the two-tenant pattern this teardown supports diff --git a/.claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh b/.claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh similarity index 99% rename from .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh rename to .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh index da1e571..53ddf4b 100644 --- a/.claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +++ b/.claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# teardown-aks-dev.sh — orchestrator for teardown-agent-aks-dev. +# teardown-aks-dev.sh — orchestrator for teardown-agent-aks-agentid. # # Safe by default: DRY_RUN=1, DELETE_ENTRA=0. # diff --git a/deploy/azure/kubernetes-service/dev/README.md b/deploy/azure/kubernetes-service/dev/README.md index 0272320..0892efa 100644 --- a/deploy/azure/kubernetes-service/dev/README.md +++ b/deploy/azure/kubernetes-service/dev/README.md @@ -21,7 +21,7 @@ In this tutorial, you learn how to: > * Verify the autonomous and on-behalf-of (OBO) identity flows end to end. > [!TIP] -> **Recommended: AI-assisted deployment.** The fastest, least error-prone way to finish this tutorial is to pair an AI assistant with the skill packaged in this repo: [`.claude/skills/deploy-agent-aks-dev/SKILL.md`](../../../.claude/skills/deploy-agent-aks-dev/SKILL.md). The assistant confirms your SKU choices, picks the right Ollama model strategy, handles the cross-tenant federation case if it applies, and surfaces known failure modes in real time — typically cutting deployment time from hours to minutes. Running the tutorial end-to-end by hand is fully supported (every command is documented below); the skill just front-loads the decisions. +> **Recommended: AI-assisted deployment.** The fastest, least error-prone way to finish this tutorial is to pair an AI assistant with the skill packaged in this repo: [`.claude/skills/deploy-agent-aks-agentid/SKILL.md`](../../../.claude/skills/deploy-agent-aks-agentid/SKILL.md). The assistant confirms your SKU choices, picks the right Ollama model strategy, handles the cross-tenant federation case if it applies, and surfaces known failure modes in real time — typically cutting deployment time from hours to minutes. Running the tutorial end-to-end by hand is fully supported (every command is documented below); the skill just front-loads the decisions. > > The skill works with **Claude Code** (which reads `.claude/skills/` by default) and with **GitHub Copilot Chat** (ask it to read the `SKILL.md` file). If you prefer a manual run, continue reading — the tutorial remains the source of truth. @@ -201,7 +201,7 @@ Before provisioning anything, pick a SKU for each of the following. The table li > [!WARNING] > **`ENABLE_LOGS=none` + Ollama init container.** The init container does `ollama pull ` on first replica start (up to 5 min for a 7B model). Without Container Insights you can only inspect this via live `kubectl logs`; once the pod restarts there is no history. Turn logs on for the first deploy. -For the full decision matrix, see the skill reference: [`sku-sizing.md`](../../../.claude/skills/deploy-agent-aks-dev/references/sku-sizing.md). +For the full decision matrix, see the skill reference: [`sku-sizing.md`](../../../.claude/skills/deploy-agent-aks-agentid/references/sku-sizing.md). ## 3. Final object inventory @@ -303,7 +303,7 @@ export CLIENT_SPA_APP_ID=$(grep '^CLIENT_SPA_APP_ID=' scripts/.env | cut -d= -f2 ### 5.3 Configure the Blueprint for OBO ```powershell -pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blueprint-for-aks.ps1 ` +pwsh -NoProfile -File .claude/skills/deploy-agent-aks-agentid/scripts/setup-obo-blueprint-for-aks.ps1 ` -BlueprintAppId $env:BLUEPRINT_APP_ID ` -ClientSpaAppId $env:CLIENT_SPA_APP_ID ` -AgentAppId $env:AGENT_CLIENT_ID ` @@ -315,7 +315,7 @@ pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/setup-obo-blue OBO requires a **delegated** `User.Read` grant in addition to the application permissions `Start-EntraAgentIDWorkflow` already granted. Without this, users hit `AADSTS65001`. ```powershell -pwsh -NoProfile -File .claude/skills/deploy-agent-aks-dev/scripts/grant-agent-obo-consent.ps1 ` +pwsh -NoProfile -File .claude/skills/deploy-agent-aks-agentid/scripts/grant-agent-obo-consent.ps1 ` -AgentAppId $env:AGENT_CLIENT_ID -TenantId $env:TENANT_ID ``` @@ -414,7 +414,7 @@ export TENANT_ID="" export SUBSCRIPTION_TENANT_ID="" ``` -Run `az login` once per tenant; the CLI tracks the two contexts side by side. The federation Graph call always uses `$TENANT_ID` (Blueprint tenant); the cluster commands always use `$SUBSCRIPTION_TENANT_ID` (Azure tenant). Full pattern, variable contract, and common errors: [`cross-tenant-federation.md`](../../../.claude/skills/deploy-agent-aks-dev/references/cross-tenant-federation.md). +Run `az login` once per tenant; the CLI tracks the two contexts side by side. The federation Graph call always uses `$TENANT_ID` (Blueprint tenant); the cluster commands always use `$SUBSCRIPTION_TENANT_ID` (Azure tenant). Full pattern, variable contract, and common errors: [`cross-tenant-federation.md`](../../../.claude/skills/deploy-agent-aks-agentid/references/cross-tenant-federation.md). ## 8. Phase 4 — Build and push container images @@ -436,14 +436,14 @@ az acr build --registry "$ACR_NAME" \ ## 9. Phase 5 — Deploy the Kubernetes workloads -The full manifest set lives in [`.claude/skills/deploy-agent-aks-dev/manifests/`](../../../.claude/skills/deploy-agent-aks-dev/manifests/) and uses `${VAR}` placeholders that `envsubst` substitutes at apply time. +The full manifest set lives in [`.claude/skills/deploy-agent-aks-agentid/manifests/`](../../../.claude/skills/deploy-agent-aks-agentid/manifests/) and uses `${VAR}` placeholders that `envsubst` substitutes at apply time. ### 9.1 Render and apply ```bash set -a; source /tmp/deploy-vars.sh; set +a # auto-export every variable -MANIFEST_DIR=".claude/skills/deploy-agent-aks-dev/manifests" +MANIFEST_DIR=".claude/skills/deploy-agent-aks-agentid/manifests" # Render with explicit varlist so typos fail loudly instead of producing empty strings VARLIST='$TENANT_ID $BLUEPRINT_APP_ID $AGENT_CLIENT_ID $CLIENT_SPA_APP_ID $ACR_NAME $OLLAMA_MODEL $STORAGE_GB' @@ -485,7 +485,7 @@ Two manual steps that can't be done before the cluster exists. ### 10.1 Add the LoadBalancer IP to the Client SPA redirect URIs ```bash -bash .claude/skills/deploy-agent-aks-dev/scripts/add-spa-redirect-uri.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/add-spa-redirect-uri.sh ``` The script PATCHes `spa.redirectUris` on the Client SPA app directly via Graph. It always adds `http://localhost:8080/` (used for the port-forward sign-in path in [§11.4](#114-obo-flow-via-port-forward)) and additionally adds `http://${APP_FQDN}/` if `APP_FQDN` is set. `az ad app update --web-redirect-uris` does **not** affect SPA redirect URIs — that's why this is a Graph PATCH. @@ -629,18 +629,18 @@ kubectl -n agentid exec deploy/ollama -- ollama list ## 15. Clean teardown -> **TIP — AI-assisted teardown.** If you use Claude Code or GitHub Copilot, invoke the [`teardown-agent-aks-dev`](../../../.claude/skills/teardown-agent-aks-dev/SKILL.md) skill. It runs the same commands below with dry-run by default and prompts at each destructive step. +> **TIP — AI-assisted teardown.** If you use Claude Code or GitHub Copilot, invoke the [`teardown-agent-aks-agentid`](../../../.claude/skills/teardown-agent-aks-agentid/SKILL.md) skill. It runs the same commands below with dry-run by default and prompts at each destructive step. > > ```bash > # Dry run (default — prints commands, deletes nothing) -> bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +> bash .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh > > # Azure only -> DRY_RUN=0 bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +> DRY_RUN=0 bash .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh > > # Full teardown (Azure + FIC + opt-in Entra apps) > DRY_RUN=0 DELETE_ENTRA=1 \ -> bash .claude/skills/teardown-agent-aks-dev/scripts/teardown-aks-dev.sh +> bash .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh > ``` ### 15.1 Order of operations @@ -696,13 +696,13 @@ Before paying for AKS, you can validate the manifest wiring on a local `kind` cl ```bash source /tmp/deploy-vars.sh export BLUEPRINT_CLIENT_SECRET="" -bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh +bash .claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh # Cleanup -bash .claude/skills/deploy-agent-aks-dev/scripts/smoke-test-kind.sh --cleanup +bash .claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh --cleanup ``` -Full details: [`smoke-test.md`](../../../.claude/skills/deploy-agent-aks-dev/references/smoke-test.md). +Full details: [`smoke-test.md`](../../../.claude/skills/deploy-agent-aks-agentid/references/smoke-test.md). ## Appendix B — Secretless migration from docker-compose From 453862a07d54ec993e3fde89a5c6ad35949d04bd Mon Sep 17 00:00:00 2001 From: vj926 Date: Thu, 11 Jun 2026 16:38:52 -0700 Subject: [PATCH 4/4] Convert AKS skill scripts to PowerShell; complete teardown coverage Addresses review feedback on PR #28: - Replace mixed bash/PowerShell with PowerShell-only - Rewrite teardown to remove all Entra objects created by setup - Clean up SPA redirect URIs and k8s namespace during teardown Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../skills/deploy-agent-aks-agentid/SKILL.md | 91 ++--- .../scripts/01-create-aks.ps1 | 69 ++++ .../scripts/01-create-aks.sh | 47 --- .../scripts/02-build-and-push.ps1 | 43 +++ .../scripts/02-build-and-push.sh | 42 --- .../scripts/04-apply-manifests.ps1 | 62 ++++ .../scripts/04-apply-manifests.sh | 48 --- .../scripts/add-spa-redirect-uri.ps1 | 60 ++++ .../scripts/add-spa-redirect-uri.sh | 83 ----- .../scripts/deploy-aks-dev.ps1 | 98 ++++++ .../scripts/deploy-aks-dev.sh | 82 ----- ...s.sh.template => deploy-vars.ps1.template} | 80 ++--- .../scripts/port-forward.ps1 | 39 +++ .../scripts/port-forward.sh | 41 --- .../scripts/smoke-test-kind.ps1 | 171 +++++++++ .../scripts/smoke-test-kind.sh | 145 -------- .../teardown-agent-aks-agentid/SKILL.md | 84 ++--- .../scripts/teardown-aks-dev.ps1 | 330 ++++++++++++++++++ .../scripts/teardown-aks-dev.sh | 234 ------------- deploy/azure/kubernetes-service/dev/README.md | 45 ++- 20 files changed, 1012 insertions(+), 882 deletions(-) create mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/01-create-aks.ps1 delete mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/01-create-aks.sh create mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/02-build-and-push.ps1 delete mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/02-build-and-push.sh create mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/04-apply-manifests.ps1 delete mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/04-apply-manifests.sh create mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/add-spa-redirect-uri.ps1 delete mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/add-spa-redirect-uri.sh create mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/deploy-aks-dev.ps1 delete mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/deploy-aks-dev.sh rename .claude/skills/deploy-agent-aks-agentid/scripts/{deploy-vars.sh.template => deploy-vars.ps1.template} (60%) create mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/port-forward.ps1 delete mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/port-forward.sh create mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.ps1 delete mode 100644 .claude/skills/deploy-agent-aks-agentid/scripts/smoke-test-kind.sh create mode 100644 .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.ps1 delete mode 100644 .claude/skills/teardown-agent-aks-agentid/scripts/teardown-aks-dev.sh diff --git a/.claude/skills/deploy-agent-aks-agentid/SKILL.md b/.claude/skills/deploy-agent-aks-agentid/SKILL.md index 7dd4740..62f657e 100644 --- a/.claude/skills/deploy-agent-aks-agentid/SKILL.md +++ b/.claude/skills/deploy-agent-aks-agentid/SKILL.md @@ -35,16 +35,15 @@ One federation chain — Kubernetes ServiceAccount → Blueprint app. **No clien - `az` ≥ 2.60 with `aks-preview` extension (`az extension add --name aks-preview`) - `kubectl` ≥ 1.28 - `pwsh` 7.4+ with `Microsoft.Graph.Authentication` (`Install-Module Microsoft.Graph.Authentication -Scope CurrentUser`) - - `envsubst` (from the `gettext` package; on Windows comes with Git Bash) - Optional for local smoke test: Docker Desktop + `kind` ≥ 0.20 - Optional for "Adapt for your own agent": a container image of the user's agent in any registry reachable by AKS 4. **Tenant + subscription confirmed with the user.** ALWAYS confirm before any `az` command that mutates resources. Users frequently have multiple tenants; pick the wrong one and you create a half-deployed cluster in the wrong place. 5. **Entra Agent ID base objects exist** — Blueprint, Agent Identity, and (for OBO) a Client SPA. If not, chain `entra-agent-id-setup` first. 6. **Resource providers registered** on first use of a fresh subscription: - `Microsoft.ContainerService`, `Microsoft.ContainerRegistry`, `Microsoft.Compute`, `Microsoft.Network`, `Microsoft.Storage`, `Microsoft.OperationalInsights`, `Microsoft.OperationsManagement`. `01-create-aks.sh` checks and registers what's missing. + `Microsoft.ContainerService`, `Microsoft.ContainerRegistry`, `Microsoft.Compute`, `Microsoft.Network`, `Microsoft.Storage`, `Microsoft.OperationalInsights`, `Microsoft.OperationsManagement`. `01-create-aks.ps1` checks and registers what's missing. > [!NOTE] -> **Windows / PowerShell users:** the orchestrator and scripts are bash + `pwsh`. Run them from **Git Bash** or **WSL**, not raw PowerShell — `source`, `envsubst`, and curl-style heredocs do not have native PowerShell equivalents. +> **All scripts are PowerShell (pwsh 7.4+).** Run them with `pwsh -NoProfile -File