-
Notifications
You must be signed in to change notification settings - Fork 29
Ocp admin/multi cluster report - automatic authentication mechanism #27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
9eff8b4
add the flag so it can read the owner-ony files
ikrispin 3b3c7b1
create an automatic multi-cluster authentication mechanism via SA tok…
ikrispin 765465d
add explicit rule to never display tokens in AI conversations
ikrispin fa0b9a1
docs: clarified the admin permission level as a prerequisite
ikrispin e872bc5
extrat python setup and build scripts
ikrispin 87e990f
refactor: consolidate scripts into a single python script
ikrispin 3565def
Rename plugin from 'OpenShift Administration Agentic Skills Collectio…
ikrispin 634969e
add skip-rbac flag for when it's already applied for extracting exist…
ikrispin e791d36
add pre-flight checks for fail-fast in case of missing prerequisites
ikrispin 26d4a87
update find_kube_cmd() to prefer oc and warn on kubectl fallback
ikrispin f8ce472
add a bash wrapper to podman to conditionally add --userns only on Linux
ikrispin 2866a56
Merge remote-tracking branch 'origin' into ocp-admin/multi-cluster-re…
ikrispin 35e40fa
Merge branch 'ocp-admin/multi-cluster-report' of github.com:RHEcosyst…
ikrispin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,248 @@ | ||
| # Multi-Cluster Authentication with Service Account Tokens | ||
|
|
||
| Set up non-interactive, long-lived authentication for running `cluster-report` across many OpenShift clusters without repeated `oc login` sessions. | ||
|
|
||
| ## Overview | ||
|
|
||
| The `cluster-report` skill requires valid kubeconfig contexts for every cluster it reports on. Interactive `oc login --web` opens a browser for each cluster and produces tokens that expire in ~24 hours which make it difficult to do at scale. | ||
|
|
||
| **Solution**: Create a read-only ServiceAccount on each cluster with a non-expiring token. A builder script assembles these tokens into a single merged kubeconfig that the skill uses unchanged. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - `oc` or `kubectl` CLI | ||
| - `python3` (stdlib only, no extra packages) | ||
| - `cluster-admin` access on each target cluster (one-time setup only) | ||
|
|
||
| ## Quick Start (Automated) | ||
|
|
||
| If you're currently logged into all the clusters you would like to get a report for via `oc login`: | ||
|
|
||
| ```bash | ||
| # Step 1: Setup — applies RBAC to each cluster, extracts SA tokens | ||
| python3 ocp-admin/scripts/cluster-report/build-kubeconfig.py setup --all-contexts | ||
|
|
||
| # Step 2: Build — assembles a merged kubeconfig from the inventory | ||
| python3 ocp-admin/scripts/cluster-report/build-kubeconfig.py \ | ||
| build --clusters ~/.ocp-clusters/clusters.json --verify | ||
|
|
||
| # Step 3: Use — export and run the skill | ||
| export KUBECONFIG=/tmp/cluster-report-kubeconfig | ||
| # Then in Claude Code use the skill: /cluster-report | ||
| ``` | ||
|
|
||
| After the one-time setup, only Steps 2–3 are needed for future report sessions. | ||
|
|
||
| ## Manual Setup (Per Cluster) | ||
|
|
||
| If you prefer to set up each cluster individually: | ||
|
|
||
| ### 1. Apply RBAC | ||
|
|
||
| > **Required permissions**: The manifest creates cluster-scoped resources (ClusterRole, ClusterRoleBinding), so the user applying it needs `cluster-admin` privileges. This is a one-time setup step. | ||
|
|
||
| ```bash | ||
| oc login <cluster-api-url> | ||
| oc apply -f ocp-admin/scripts/cluster-report/cluster-reporter-rbac.yaml | ||
| ``` | ||
|
|
||
| This creates: | ||
|
|
||
| - Namespace `cluster-reporter-system` | ||
| - ServiceAccount `cluster-reporter` with a read-only ClusterRole | ||
| - ClusterRoleBinding `cluster-reporter-binding` (binds the SA to the ClusterRole) | ||
| - Token Secret `cluster-reporter-token` (non-expiring) | ||
|
|
||
| ### 2. Extract the Token | ||
|
|
||
| ```bash | ||
| oc get secret cluster-reporter-token -n cluster-reporter-system \ | ||
| -o jsonpath='{.data.token}' | base64 -d | ||
| ``` | ||
|
|
||
| Save this token securely. It grants read-only access to nodes, pods, namespaces, projects, cluster version, and metrics. | ||
|
dmartinol marked this conversation as resolved.
|
||
|
|
||
| > **AI Safety**: Never display token values in conversation output. Verify tokens are set, but never print or echo their contents. | ||
|
|
||
| ### 3. Add to Inventory File | ||
|
|
||
| Create or edit `~/.ocp-clusters/clusters.json`: | ||
|
|
||
| ```json | ||
| { | ||
| "clusters": [ | ||
| { | ||
| "name": "prod-us-east", | ||
| "api_url": "https://api.prod-us-east.example.com:6443", | ||
| "token": "sha256~your-token-here" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Set permissions: `chmod 600 ~/.ocp-clusters/clusters.json` | ||
|
|
||
| ### 4. Build Kubeconfig | ||
|
|
||
| ```bash | ||
| python3 ocp-admin/scripts/cluster-report/build-kubeconfig.py \ | ||
| build --clusters ~/.ocp-clusters/clusters.json --output ~/.kube/cluster-report-kubeconfig | ||
| ``` | ||
|
|
||
| ## RBAC Permissions | ||
|
|
||
| The `cluster-reporter-readonly` ClusterRole grants the minimum permissions required by the `cluster-report` skill: | ||
|
|
||
|
|
||
| | Resource | API Group | Verbs | Used By | | ||
| | ----------------------- | -------------------- | --------- | ------------------------------------------------------------- | | ||
| | nodes, namespaces, pods | core | get, list | `nodes_top`, `resources_list`, `namespaces_list`, `pods_list` | | ||
| | clusterversions | config.openshift.io | get | `resources_get` (OpenShift verification) | | ||
| | projects | project.openshift.io | list | `projects_list` | | ||
| | nodes, pods (metrics) | metrics.k8s.io | get, list | `nodes_top` | | ||
|
|
||
|
|
||
| No create, update, delete, or watch permissions are granted. | ||
|
|
||
| ## Clusters Inventory Format | ||
|
|
||
| The inventory file (`clusters.json`) supports two token modes: | ||
|
|
||
| ### Inline Tokens (Simple) | ||
|
|
||
| ```json | ||
| { | ||
| "clusters": [ | ||
| { | ||
| "name": "prod-us-east", | ||
| "api_url": "https://api.prod-us-east.example.com:6443", | ||
| "token": "sha256~abc123..." | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| The file itself contains secrets — keep it out of git and set `chmod 600`. | ||
|
|
||
| ### Environment Variable References (More Secure) | ||
|
|
||
| ```json | ||
| { | ||
| "clusters": [ | ||
| { | ||
| "name": "prod-us-east", | ||
| "api_url": "https://api.prod-us-east.example.com:6443", | ||
| "token_env": "CLUSTER_TOKEN_PROD_US_EAST" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| The file contains no secrets. Load tokens into environment variables from your secrets manager before running `--build`. | ||
|
|
||
| ### Optional: CA Certificate | ||
|
|
||
| ```json | ||
| { | ||
| "clusters": [ | ||
| { | ||
| "name": "prod-us-east", | ||
| "api_url": "https://api.prod-us-east.example.com:6443", | ||
| "token": "sha256~abc123...", | ||
| "ca_cert": "/path/to/prod-us-east-ca.crt" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| If `ca_cert` is omitted, TLS verification is skipped (`--insecure-skip-tls-verify`). | ||
|
|
||
| ## Script Reference | ||
|
|
||
| ### `setup` Subcommand | ||
|
|
||
| ```bash | ||
| python3 build-kubeconfig.py setup [OPTIONS] | ||
| ``` | ||
|
|
||
|
|
||
| | Flag | Description | Default | | ||
| | --------------------------- | ----------------------------- | ------------------------------- | | ||
| | `--all-contexts` | Setup all kubeconfig contexts | Lists contexts and exits | | ||
| | `--contexts ctx1,ctx2` | Setup only specified contexts | — | | ||
| | `--output-inventory <path>` | Inventory file path | `~/.ocp-clusters/clusters.json` | | ||
|
|
||
|
|
||
| Behavior: | ||
|
|
||
| - Applies `cluster-reporter-rbac.yaml` to each cluster | ||
| - Waits up to 15 seconds for the token Secret to populate | ||
| - Extracts and saves the token to the inventory file | ||
| - Skips unreachable clusters with an error message | ||
| - Appends to existing inventory (deduplicates by name) | ||
|
|
||
| ### `build` Subcommand | ||
|
|
||
| ```bash | ||
| python3 build-kubeconfig.py build --clusters <path> [OPTIONS] | ||
| ``` | ||
|
|
||
|
|
||
| | Flag | Description | Default | | ||
| | ------------------- | -------------------------------- | -------------------------------- | | ||
| | `--clusters <path>` | Inventory file path (required) | — | | ||
| | `--output <path>` | Kubeconfig output path | `/tmp/cluster-report-kubeconfig` | | ||
| | `--verify` | Test each context after building | Off | | ||
|
|
||
|
|
||
| Behavior: | ||
|
|
||
| - Reads inventory, resolves tokens (inline or env var) | ||
| - Builds kubeconfig with `kubectl config set-cluster/set-credentials/set-context` | ||
| - Partial success: continues on individual failures | ||
| - `--verify` tests each context with `cluster-info` | ||
| - Outputs JSON summary with success/error counts | ||
|
|
||
| ## Token Rotation | ||
|
|
||
| SA token Secrets do not expire, but you may want to rotate them periodically: | ||
|
|
||
| ```bash | ||
| oc delete secret cluster-reporter-token -n cluster-reporter-system | ||
| oc apply -f ocp-admin/scripts/cluster-report/cluster-reporter-rbac.yaml | ||
|
|
||
| oc get secret cluster-reporter-token -n cluster-reporter-system \ | ||
| -o jsonpath='{.data.token}' | base64 -d | ||
|
|
||
| python3 build-kubeconfig.py build --clusters ~/.ocp-clusters/clusters.json --verify | ||
| ``` | ||
|
|
||
| To detect expired or invalid tokens: | ||
|
|
||
| ```bash | ||
| python3 build-kubeconfig.py build --clusters ~/.ocp-clusters/clusters.json --verify | ||
| ``` | ||
|
|
||
| ## Security Best Practices | ||
|
|
||
| 1. **Never commit tokens to git** — add `clusters.json` to `.gitignore` | ||
| 2. **File permissions** — `chmod 600` on both `clusters.json` and the generated kubeconfig | ||
| 3. **Prefer `token_env`** — store actual tokens in a secrets manager, not in files | ||
| 4. **Minimum RBAC** — the ClusterRole grants read-only access only | ||
| 5. **Dedicated namespace** — the SA lives in `cluster-reporter-system`, not `kube-system` | ||
| 6. **Generated kubeconfig is ephemeral** — `/tmp/` is fine for session use; for persistent storage use `~/.kube/` with `chmod 600` | ||
| 7. **Never display tokens in AI conversations** — verify tokens are set but never print, echo, or expose their values in output | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
|
|
||
| | Problem | Cause | Fix | | ||
| | ---------------------------------------- | ----------------------------------------- | ------------------------------------------------------------- | | ||
| | `--setup` skips a cluster | Not logged in or auth expired | `oc login <api-url>` first, then re-run setup | | ||
| | `--verify` fails for a cluster | Token expired or Secret deleted | Re-run `--setup --contexts <ctx>` for that cluster | | ||
| | `cluster-report` shows 401 for a cluster | Token invalid | Same as above — re-run setup for that cluster | | ||
| | `cluster-report` shows 403 | SA missing permissions | Re-apply `cluster-reporter-rbac.yaml` on that cluster | | ||
| | Token Secret not populated | Token controller slow or SA doesn't exist | Wait and retry; verify SA exists in `cluster-reporter-system` | | ||
| | `--build` says "env var not set" | Using `token_env` but env not loaded | Export the token env vars before running `--build` | | ||
|
|
||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.