From 0d354653dfe9accc0ce42713cc4d534e0d2819e7 Mon Sep 17 00:00:00 2001 From: Eyal Bukchin Date: Thu, 18 Jun 2026 11:32:34 -0400 Subject: [PATCH] Revert docs#214 heading demotion on deeply-nested pages (restore anchors) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs#214 demoted body headings by one level to remove a duplicate H1. On these 10 deeply-nested pages that pushed real H3 subheadings to H4, and GitBook only generates anchor links for H1-H3 — so they silently lost their deep-link anchors and dropped out of the page outline. Un-demote those pages (exact inverse: remove one '#' per heading, code fences untouched), restoring the H3-level anchors. Descriptions and other content are preserved. The reinstated body '#' H1 re-creates the "multiple H1" the audit flagged (Trivial) — accepted, since GitBook's page-title-is-H1 model makes single-H1 and fully-anchored multi-level content mutually exclusive here, and anchors matter more. (quick-start and preview-environments were shallow and keep the #214 demotion.) Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/managing-mirrord/admin-dashboard.md | 26 ++-- docs/managing-mirrord/license-server.md | 14 +-- docs/managing-mirrord/monitoring.md | 28 ++--- docs/managing-mirrord/operator.md | 20 +-- .../db-branching-advanced-config.md | 54 ++++----- docs/sharing-the-cluster/queue-splitting.md | 114 +++++++++--------- docs/troubleshooting/utilities.md | 14 +-- docs/use-cases/mirrord-for-ci.md | 14 +-- docs/using-mirrord/multi-cluster-setup.md | 80 ++++++------ .../multiple-concurrent-sessions.md | 34 +++--- 10 files changed, 199 insertions(+), 199 deletions(-) diff --git a/docs/managing-mirrord/admin-dashboard.md b/docs/managing-mirrord/admin-dashboard.md index c6d19d92..a60fe704 100644 --- a/docs/managing-mirrord/admin-dashboard.md +++ b/docs/managing-mirrord/admin-dashboard.md @@ -15,7 +15,7 @@ tags: description: Dashboard for monitoring mirrord usage --- -## Dashboard +# Dashboard The mirrord Dashboard is a web-based interface for monitoring mirrord usage across your organization. It provides real-time visibility into sessions, users, targets, CI pipelines, and overall adoption trends, all served directly from the license server. @@ -31,7 +31,7 @@ Want to see the dashboard in action? Check out our [live playground](https://pla |-----------|------------| | ![Dashboard - Dark Mode](admin-dashboard/images/dashboard-dark.png) | ![Dashboard - Light Mode](admin-dashboard/images/dashboard-light.png) | -### Quick Start +## Quick Start 1. Add `dashboard.enabled: true` to your license server Helm values: @@ -62,11 +62,11 @@ The dashboard reads from the license server's existing session database, so your The dashboard does not require authentication beyond network access to the license server. Access control is handled by your cluster networking configuration. {% endhint %} -### Usage Tab +## Usage Tab The Usage tab is the main view, showing metrics, session activity, and analytics charts. -#### Metric Cards and Session Activity +### Metric Cards and Session Activity ![Metric cards and session activity table](admin-dashboard/images/metrics-and-activity.png) @@ -84,7 +84,7 @@ Use the time range selector in the top-right corner (**7d**, **30d**, **90d**, * Below the metrics, the **Session Activity** table shows a cross-referenced view of users and their target workloads. Each row shows the user, target, namespace, session count with a visual bar, and total time. The table is searchable and sortable by any column, with pagination for large datasets. -#### Users View +### Users View ![User charts and metrics table](admin-dashboard/images/users-view.png) @@ -94,7 +94,7 @@ Switch to the **Users** tab to see user-focused analytics: - **User Timeline**: Shows when each user was first seen and their most recent activity, giving a quick view of adoption over time. - **User Metrics** table: A detailed, searchable table with columns for identifier, first active date, last seen date, total sessions, cumulative time, and average duration. Click any column header to sort. -#### Targets View +### Targets View ![Target adoption and namespace breakdown](admin-dashboard/images/targets-view.png) @@ -104,7 +104,7 @@ Switch to the **Targets** tab to see target-focused analytics: - **Sessions by Namespace**: A horizontal bar chart breaking down session distribution across Kubernetes namespaces. - **Target Metrics** table: A searchable table listing each target workload with its namespace, session count, total time, and number of unique users. -### ROI Calculator +## ROI Calculator ![ROI Calculator](admin-dashboard/images/roi-calculator.png) @@ -124,21 +124,21 @@ The **ROI Calculator** tab estimates the time and cost savings from using mirror - **Annual infrastructure savings** - **Net Annual ROI** after subtracting the mirrord license cost -### Features +## Features -#### Dark Mode +### Dark Mode Toggle between light and dark themes using the moon/sun icon in the top-right corner of the app bar. Your preference is saved in the browser's local storage. -#### Manual Sync +### Manual Sync Click the **Sync** button in the app bar to manually refresh all dashboard data. The last updated timestamp is displayed next to the button. -#### Operator Version +### Operator Version The operator version is displayed in the app bar for quick reference (e.g., `v3.142.0`). -### Helm Configuration +## Helm Configuration | Setting | Default | Description | |---------|---------|-------------| @@ -147,7 +147,7 @@ The operator version is displayed in the app bar for quick reference (e.g., `v3. The chart automatically configures the container port, service port, and required environment variables when `dashboard.enabled` is set to `true`. -### API Endpoints +## API Endpoints The dashboard consumes two API endpoints from the license server. These are also available for programmatic access: diff --git a/docs/managing-mirrord/license-server.md b/docs/managing-mirrord/license-server.md index 5bee4ab3..097b494a 100644 --- a/docs/managing-mirrord/license-server.md +++ b/docs/managing-mirrord/license-server.md @@ -21,13 +21,13 @@ The license server enables you to manage your organization’s seats without sen This feature is available to users on the Enterprise pricing plan. {% endhint %} -#### Architecture +### Architecture The license server exposes an API that the operator uses to obtain a license, which the operator requires to run. It also allows the operator to keep track of active users, check and enforce seat counts, and store telemetry data. Authentication between the operator and license server is done using a license key (chosen by the person setting up the license server) and is not to be confused with the license certificate file provided by MetalBear. -#### Basic Setup +### Basic Setup The license server is installable via Helm. First, add the MetalBear Helm repository: @@ -77,7 +77,7 @@ kubectl get deployment -n mirrord mirrord-license-server If your operator(s) are running in a different cluster, make sure the `mirrord-operator-license-server` service is exposed to them via ingress. -##### Using a Cluster Secret +#### Using a Cluster Secret You can set the license key in a cluster secret within the operator's namespace (`mirrord` by default), and reference it in the license server helm chart via `license.keyRef`. For example, with the following `values.yaml`: @@ -104,7 +104,7 @@ The secret itself, which must use the key `OPERATOR_LICENSE_KEY`, can be created kubectl create secret generic my-cluster-secret -n mirrord --from-literal OPERATOR_LICENSE_KEY=my-very-secret-string ``` -##### Using Google Secrets Manager +#### Using Google Secrets Manager You can fetch the license file from GSM by providing the secret path and service account credentials as follows: @@ -129,7 +129,7 @@ sa: gcpSa: @.iam.gserviceaccount.com ``` -##### Connecting Operators to the License Server +#### Connecting Operators to the License Server First update your operator `values.yaml` file ([see this page](../getting-started/quick-start.md#helm) for quickstart helm setup for operator): @@ -148,7 +148,7 @@ Then run: helm install metalbear/mirrord-operator -f ./values.yaml --wait ``` -### Getting a Utilisation Report from the License Server +## Getting a Utilisation Report from the License Server {% hint style="info" %} This feature requires at least mirrord-operator-license-server Helm chart version **1.4.0**. @@ -186,7 +186,7 @@ To get a report: mirrord exec -- curl "mirrord-operator-license-server.mirrord.svc.cluster.local/api/v1/reports/usage?format=xlsx" --header 'x-license-key: ' --output report.xlsx ``` -### Getting Operator Failure Reports from the License Server +## Getting Operator Failure Reports from the License Server {% hint style="info" %} This feature requires at least mirrord-operator-license-server Helm chart version **1.56.0**. diff --git a/docs/managing-mirrord/monitoring.md b/docs/managing-mirrord/monitoring.md index e1ace556..62c4a5cd 100644 --- a/docs/managing-mirrord/monitoring.md +++ b/docs/managing-mirrord/monitoring.md @@ -22,7 +22,7 @@ The mirrord Operator can produce logs in JSON format that can be digested by mos This feature is available to users on the Team and Enterprise pricing plans. {% endhint %} -### Functional Logs +## Functional Logs The following logs are written with log level `INFO`, and can be used for dashboards within monitoring solutions in order to monitor mirrord usage within your organization: @@ -51,11 +51,11 @@ Fields: | http\_filter | the client's configured [HTTP Filter](https://metalbear.com/mirrord/docs/config#feature.network) | `Port Steal` | | scale\_down | whether the session's target was scaled down | `Copy Target` | -### Prometheus +## Prometheus The mirrord Operator can expose Prometheus metrics if enabled (the default endpoint is `:9000/metrics`). -#### Helm +### Helm ```yaml # values.yaml for mirrord-operator helm chart @@ -65,14 +65,14 @@ operator: ... ``` -#### Manual +### Manual | env | description | type | default | | -------------------------- | ------------------------ | ----------------- | -------------- | | OPERATOR\_METRICS\_ENABLED | enable metrics endpoint | "true" \| "false" | "false" | | OPERATOR\_METRICS\_ADDR | metrics http server addr | SocketAddr | "[::]:9000" | -#### Exposed metrics +### Exposed metrics | metric | description | labels | minimum version | | -------------------------------- | ---------------------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------- | @@ -90,13 +90,13 @@ operator: | mirrord_previews_create_total | Count of created preview sessions | `target_namespace` `target_kind` `target_name` | operator 3.163.0 | | mirrord_previews_duration | Histogram for finished preview sessions duration | `target_namespace` `target_kind` `target_name` | operator 3.163.0 | -### OpenTelemetry +## OpenTelemetry {% hint style="info" %} The features under the "OpenTelemetry" heading require at least operator chart version 1.46.0. {% endhint %} -#### Exporting Logs +### Exporting Logs To export logs from the operator to an endpoint, set `operator.otelLogExportUrl` to the URL in the Operator Helm chart values. You _must_ set this value to export logs. This value does not affect the logs which are printed by the operator to `stdout` and are always enabled. @@ -104,11 +104,11 @@ The log level is `INFO` by default, and can be changed by setting `operator.otel Note that this log level is separate to that defined for logs controlled by `operator.logLevel`, which are printed by the operator to `stdout`. -#### Exporting Traces +### Exporting Traces To export traces from the operator to an endpoint, set `operator.otelTraceExportUrl` to the URL in the Operator Helm chart values. You _must_ set this value to export traces. -#### Context Propagation +### Context Propagation {% hint style="info" %} This feature requires at least mirrord version 3.184.0. @@ -129,21 +129,21 @@ The Operator will propagate these values into exported spans for some frequently For more info about using `traceparent` and `baggage`, see [the OpenTelemetry docs about context propagation](https://opentelemetry.io/docs/concepts/context-propagation/). -### Pre-Built Dashboards +## Pre-Built Dashboards -#### DataDog Dashboard +### DataDog Dashboard We offer a DataDog dashboard you can import to track statistics. Download it [here](https://github.com/metalbear-co/docs/tree/main/docs/managing-mirrord/assets/Mirrord_datadog_Operator_Dashboard.json). -#### Grafana Dashboard +### Grafana Dashboard Alternatively there is a Grafana dashboard you can import to track statistics. Download it [here](https://github.com/metalbear-co/docs/tree/main/docs/managing-mirrord/assets/Mirrord_grafana_Operator_Dashboard.json). -### fluentd +## fluentd If you are using fluentd you can add a filter to unpack some values from the "log" message: @@ -161,7 +161,7 @@ If you are using fluentd you can add a filter to unpack some values from the "lo This will expand all the extra fields stored in the "log" field. -#### fluentd + Elasticsearch +### fluentd + Elasticsearch Assuming you are using `logstash_format true` and the connected mapping will store the extra fields in a `keyword` type, we have a ready made dashboard you can simply import. diff --git a/docs/managing-mirrord/operator.md b/docs/managing-mirrord/operator.md index 05b01631..408e44eb 100644 --- a/docs/managing-mirrord/operator.md +++ b/docs/managing-mirrord/operator.md @@ -8,7 +8,7 @@ tags: The mirrord Operator is a Kubernetes operator that runs persistently in your cluster and manages mirrord sessions. It's the central component that enables all **[Teams]** features. -### Why the Operator? +## Why the Operator? In the open-source version of mirrord, each session is standalone - mirrord injects itself into the local process and creates an agent pod directly. This works well for individual use, but doesn't support coordination between users. @@ -20,11 +20,11 @@ The Operator solves this by acting as a centralized control plane: ![mirrord for Teams - Architecture](/docs/overview/teams/operator-architecture.svg) -### Installation +## Installation You'll need a mirrord for Teams license. [Register here](https://app.metalbear.com) to get started. -#### Helm +### Helm Add the MetalBear Helm repository: @@ -79,11 +79,11 @@ Then install: helm install -f values.yaml mirrord-operator metalbear/mirrord-operator ``` -#### Using an Internal Registry (Optional) +### Using an Internal Registry (Optional) Using an internal registry reduces startup time, ingress costs, and removes dependency on GitHub's registry. -##### Feature-specific images +#### Feature-specific images These images are only pulled when the corresponding feature is enabled: @@ -92,7 +92,7 @@ These images are only pulled when the corresponding feature is enabled: | Kafka splitting sidecar | `ghcr.io/metalbear-co/operator-kafka-proxy` | Same as operator | JVM sidecar for Kafka splitting (only when `operator.kafkaSplittingSidecar.enabled` is true). | `operator.kafkaSplittingSidecar.image` | | MSSQL tools | `ghcr.io/metalbear-co/mssql-tools` | `latest` | Sidecar for MSSQL DB branching (provides `sqlcmd`, `sqlpackage`, `bcp`). | Env `MSSQL_TOOLS_IMAGE` via `operator.extraEnv` | -##### DB branching default database images +#### DB branching default database images DB branch pods pull a database image matching the engine. These are the defaults when no custom image is specified in the branch config: @@ -104,7 +104,7 @@ DB branch pods pull a database image matching the engine. These are the defaults | MSSQL | `mcr.microsoft.com/mssql/server:{version}` | `operator.mssqlBranchConfig` - `dbPod.image` | | Redis | `docker.io/library/redis:{version}` | `operator.redisBranchConfig` - `dbPod.image` | -##### Copying images +#### Copying images We recommend [regctl](https://regclient.org/) for copying multi-arch images: @@ -130,7 +130,7 @@ agent: registry: your-registry/mirrord ``` -#### OpenShift +### OpenShift Apply the following SecurityContextConstraints: @@ -152,7 +152,7 @@ users: - system:serviceaccount:mirrord:default ``` -#### GKE Autopilot +### GKE Autopilot In GKE Autopilot the mirrord Operator can be run as a [customer-owned privileged workload](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-autopilot-privileged-workloads#customer-owned-privileged-workloads). @@ -212,7 +212,7 @@ agent: cloud.google.com/generate-allowlist: "true" ``` -### Verifying the Installation +## Verifying the Installation ```bash mirrord operator status diff --git a/docs/sharing-the-cluster/db-branching-advanced-config.md b/docs/sharing-the-cluster/db-branching-advanced-config.md index 29c23def..80cd69ef 100644 --- a/docs/sharing-the-cluster/db-branching-advanced-config.md +++ b/docs/sharing-the-cluster/db-branching-advanced-config.md @@ -37,7 +37,7 @@ These settings give additional flexibility in how mirrord handles database branc } ``` -## Branch Creation Timeout +# Branch Creation Timeout `creation_timeout_secs` Defines how long (in seconds) mirrord waits for a database branch to become ready after creation. @@ -45,11 +45,11 @@ If the branch isn’t ready within this time, mirrord session fails, exists and Use this field to avoid hanging operations when branch creation takes too long or fails. Default value is 60 seconds. -## Connection Modes +# Connection Modes mirrord supports two ways of specifying how to connect to the source database: a full **connection URL** or **individual connection parameters**. -#### Connection URL +### Connection URL Provide a single environment variable that contains the full database connection string: @@ -66,7 +66,7 @@ The optional `type` field controls where the environment variable is read from ( - `"env"` (default): Direct `env` entry in the target pod spec. - `"env_from"`: From the target pod's `envFrom` field (`secretRef` or `configMapRef`). mirrord replicates the `envFrom` sources onto the init container so it can resolve the variable at runtime. -#### Individual Connection Parameters (Params) +### Individual Connection Parameters (Params) Instead of a single connection URL, you can specify each connection parameter separately. This is useful when your application stores host, port, user, password, and database as individual environment variables. @@ -86,7 +86,7 @@ Available parameters: `host`, `port`, `user`, `password`, `database`. Each field } ``` -#### Secret Source +### Secret Source Any individual connection parameter can be sourced directly from a Kubernetes Secret instead of an environment variable. This is useful when credentials are stored in Kubernetes Secrets, such as AWS Secrets Manager synced secrets or volume-mounted secret files. @@ -114,7 +114,7 @@ In this example, `host` and `database` are read from environment variables, whil The `secret` source is only supported for individual connection parameters, not for the full connection URL. {% endhint %} -#### Literal Value +### Literal Value You can provide a connection parameter as a literal value directly in the config. This is useful when the credential is injected at runtime by an external system and does not appear in the pod spec where mirrord can read it. @@ -136,7 +136,7 @@ Use a field with `value`: Works for any connection parameter (`host`, `port`, `user`, `password`, `database`). The CLI stores the literal value in a Kubernetes Secret. The operator uses it to connect the branch DB to the source and also injects it under the name you set in `env_var_name` for your local process, so your code can read it with `os.Getenv(...)` (or equivalent) even when the target pod doesn't expose it. -#### Composite Environment Variables +### Composite Environment Variables Some applications pack multiple connection details into a single environment variable. For example, a target pod might expose: @@ -169,14 +169,14 @@ Here `host` and `port` live inside the same `DB_SERVER` value. Use `value_patter During a session, only the matched part of the value is swapped out: just the host, or just the port. The rest of the string always stays intact, so your app still sees `DB_SERVER` in the `host:port` format it expects. -#### Choosing the capture group +### Choosing the capture group The capture group name follows the parameter name - `(?P...)` for the `host` variable, `(?P...)` for the `port` variable. For single-parameter patterns you can also use `(?P...)` as a generic name, or a plain unnamed group like ([^:]+). If the regex contains more than one unnamed group, the first one is used. > The regex must contain at least one capture group, otherwise the configuration is rejected. -#### Multiple Sources for the Same Parameter +### Multiple Sources for the Same Parameter Both `url` and individual `params` fields accept either a single value or an array. This is useful when an application uses several env vars for the same logical connection. For example, separate read/write URLs. @@ -193,7 +193,7 @@ Both `url` and individual `params` fields accept either a single value or an arr The **first entry** is used to locate the source database and clone it. During the session, **every entry** is rewritten to point at the branch pod. In the example above, `DATABASE_WRITE_URL` is read to find the source database, but both `DATABASE_WRITE_URL` and `DATABASE_READ_URL` are redirected to the branch, so the application reads and writes against the same branch instead of pointing reads at the original database. -#### Combining arrays with `value_pattern` +### Combining arrays with `value_pattern` If the same connection parameter appears in multiple env vars and each var encodes a composite value, use an array of `value_pattern` objects. As with plain arrays, the first entry is used as the source. Even if `WRITE_SERVER` and `READ_SERVER` point to different databases, only `WRITE_SERVER` is cloned. During the session, all entries are rewritten to point at the branch. @@ -222,7 +222,7 @@ For example, when both `WRITE_SERVER` and `READ_SERVER ` hold a `host:port` pair The same rule applies: `WRITE_SERVER` (the first entry) is used to extract the source connection. During the session, all entries - `WRITE_SERVER`, `READ_SERVER`, both user vars, and both password vars - are rewritten to point at the branch. -## Copy Modes (MySQL, PostgreSQL & MSSQL) +# Copy Modes (MySQL, PostgreSQL & MSSQL) The `copy` field controls what data gets cloned when creating a database branch. The following modes apply to MySQL, PostgreSQL, and MSSQL. For MongoDB copy modes, see [MongoDB Copy Modes](#mongodb-copy-modes) below. @@ -266,7 +266,7 @@ Developers can customize what gets copied per table. This allows copying only sp } ``` -##### In this example +#### In this example The schema for all tables is cloned. The `users` table copy includes only rows for `alice` and `bob`. @@ -277,7 +277,7 @@ Filtering can also be combined with `"mode": "empty"`, in which case only the sp Note: Filtering is not compatible with `"mode": "all"`. If both are specified, mirrord ignores the `tables` configuration. -## MongoDB Copy Modes +# MongoDB Copy Modes MongoDB supports two copy modes. The copy mode sets the **default behavior** for all collections. When combined with [collection filters](#mongodb-collection-filters), the mode determines what happens to collections that are _not_ listed in the filter. Filtered collections always receive only the matching documents. @@ -305,7 +305,7 @@ Copying large datasets can significantly increase branch creation time and stora MongoDB does not support a `"schema"` copy mode. In relational databases, `"schema"` copies table structures without data. MongoDB is schema-less and collections don't have a predefined structure separate from their documents, so `"empty"` and `"all"` are the available options. {% endhint %} -#### MongoDB Collection Filters +### MongoDB Collection Filters Developers can customize which collections are copied and apply MongoDB query filters per collection. @@ -316,7 +316,7 @@ The copy mode controls the **baseline** (what happens to collections not mention | `"empty"` | Not created | Created with matching documents only | | `"all"` | Fully copied (all documents) | Copied with matching documents only | -##### Example: `"mode": "all"` with filters +#### Example: `"mode": "all"` with filters ```json { @@ -336,7 +336,7 @@ The copy mode controls the **baseline** (what happens to collections not mention All collections are copied, but the `users` collection includes only documents for alice and bob, and the `orders` collection includes only documents created after the given timestamp. -##### Example: `"mode": "empty"` with filters +#### Example: `"mode": "empty"` with filters ```json { @@ -353,7 +353,7 @@ All collections are copied, but the `users` collection includes only documents f Only the `users` collection is created, containing documents for alice and bob. All other collections are not created. This is useful when you only need a subset of reference data and your application handles the rest through migrations. -## Redis Copy Modes +# Redis Copy Modes Redis copy modes apply only to **remote** Redis branches. A local Redis branch always starts empty. Redis supports two modes: @@ -382,7 +382,7 @@ In this example, only keys prefixed with `user:` or `session:` are copied; every Redis has no schema, so there is no `"schema"` copy mode - only `"empty"` and `"all"` are available. Filtering is done by key pattern rather than by SQL query. {% endhint %} -## IAM Authentication +# IAM Authentication mirrord supports IAM authentication for **AWS RDS** and **GCP Cloud SQL**. Credentials are read from the **target pod's environment**, just like connection URLs. @@ -390,9 +390,9 @@ mirrord supports IAM authentication for **AWS RDS** and **GCP Cloud SQL**. Crede **Default environment variables**: If you do not specify custom credential sources, mirrord automatically looks for standard environment variables in the target pod (e.g., `AWS_REGION`, `GOOGLE_APPLICATION_CREDENTIALS`). You only need additional configuration if your pod uses non-standard variable names. {% endhint %} -### AWS RDS IAM Authentication +## AWS RDS IAM Authentication -#### Minimal Configuration +### Minimal Configuration Uses the standard AWS environment variables already present in the target pod. @@ -411,7 +411,7 @@ Uses the standard AWS environment variables already present in the target pod. } ``` -#### Default AWS environment variables +### Default AWS environment variables mirrord reads the following variables from the **target pod**, not from your local shell. @@ -422,7 +422,7 @@ mirrord reads the following variables from the **target pod**, not from your loc | `secret_access_key` | `AWS_SECRET_ACCESS_KEY` | | `session_token` | `AWS_SESSION_TOKEN` | -#### Custom AWS environment variables +### Custom AWS environment variables Use this only if your pod uses non-standard variable names. @@ -437,9 +437,9 @@ Use this only if your pod uses non-standard variable names. } ``` -### GCP Cloud SQL IAM Authentication +## GCP Cloud SQL IAM Authentication -#### Minimal Configuration +### Minimal Configuration Uses the standard `GOOGLE_APPLICATION_CREDENTIALS` file path from target pod @@ -458,14 +458,14 @@ Uses the standard `GOOGLE_APPLICATION_CREDENTIALS` file path from target pod } ``` -#### Default GCP environment variables +### Default GCP environment variables | Field | Default fallback | |-------|------------------| | `credentials_path` | `GOOGLE_APPLICATION_CREDENTIALS` | | `project` | `GOOGLE_CLOUD_PROJECT`, `GCP_PROJECT`, `GCLOUD_PROJECT` | -#### Custom GCP credentials +### Custom GCP credentials You can override the default behavior in one of the following ways. @@ -498,7 +498,7 @@ Read credentials from a file path provided via an environment variable (for exam Use either `credentials_json` OR `credentials_path`, not both. {% endhint %} -##### Required Database Settings +#### Required Database Settings GCP Cloud SQL requires TLS. Make sure your `DATABASE_URL` includes: `sslmode=require` diff --git a/docs/sharing-the-cluster/queue-splitting.md b/docs/sharing-the-cluster/queue-splitting.md index 235528e9..a5466c24 100644 --- a/docs/sharing-the-cluster/queue-splitting.md +++ b/docs/sharing-the-cluster/queue-splitting.md @@ -29,7 +29,7 @@ Queue splitting is currently available for [Amazon SQS](https://aws.amazon.com/s The word "queue" in this doc is used to also refer to "topic" in the context of Kafka and Azure Service Bus, and "subscription" in the context of Google Cloud Pub/Sub. {% endhint %} -### How It Works +## How It Works When a queue splitting session starts, the mirrord operator patches the target workload (e.g. deployment or rollout) to consume messages from a different, temporary queue. That temporary queue is *exclusive* to the target workload. @@ -141,7 +141,7 @@ Please note that: 3. For Google Cloud Pub/Sub, the operator creates temporary topics and subscriptions. The target workload's subscription environment variable is patched to read from a temporary subscription, while the operator drains the original subscription and forwards messages through temporary topics. -### Enabling Queue Splitting in Your Cluster +## Enabling Queue Splitting in Your Cluster {% tabs %} @@ -150,14 +150,14 @@ Please note that: {% stepper %} {% step %} -#### Enable SQS splitting in the Helm chart +### Enable SQS splitting in the Helm chart Enable the `operator.sqsSplitting` setting in the [mirrord-operator Helm chart](https://github.com/metalbear-co/charts/blob/main/mirrord-operator/values.yaml). {% endstep %} {% step %} -#### Authenticate and authorize the mirrord operator +### Authenticate and authorize the mirrord operator The mirrord operator will need to be able to perform operations on the SQS queues. To do this, it will build an SQS client, using the default credentials provider chain. @@ -248,7 +248,7 @@ In that case, grant the operator: {% endstep %} {% step %} -#### Authorize deployed consumers +### Authorize deployed consumers In order to be targeted with SQS splitting, a deployed consumer must be able to use the temporary queues created by mirrord. E.g. if the consumer application retrieves the queue's URL based on its name, lists queue's tags, consumes and deletes messages from the queue — it must be able to do the same on a temporary queue. @@ -261,7 +261,7 @@ However, if the consumer's access to the queue is controlled by an IAM policy (a {% endstep %} {% step %} -#### Provide application context +### Provide application context On operator installation with `operator.sqsSplitting` enabled, a new [`CustomResource`](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) type is defined in your cluster — `MirrordWorkloadQueueRegistry`. Users with permissions to get CRDs can verify its existence @@ -305,7 +305,7 @@ The registry above says that: 3. The SQS queues can be referenced in a mirrord config under IDs `meme-queue` and `ad-queue`, respectively. 4. When creating a temporary queue derived from either of the two queues, mirrord operator should add the tag `tool=mirrord`. -##### Link the registry to the deployed consumer +#### Link the registry to the deployed consumer The queue registry is a namespaced resource, so it can only reference a consumer deployed in the same namespace. The reference is specified with `spec.consumer`: @@ -313,7 +313,7 @@ The reference is specified with `spec.consumer`: * `workloadType` — type of the Kubernetes workload of the deployed consumer. Right now only consumers deployed in deployments and rollouts are supported. * `container` — name of the exact container running in the workload. This field is optional. If you omit it, the registry will reference all of the workload's containers. -##### Desribe consumed queues in the registry +#### Desribe consumed queues in the registry The queue registry describes SQS queues consumed by the referenced consumer. The queues are described in entries of the `spec.queues` object. @@ -381,14 +381,14 @@ The mirrord operator can only read consumer's environment variables if they are {% stepper %} {% step %} -#### Enable Kafka splitting in the Helm chart +### Enable Kafka splitting in the Helm chart Enable the `operator.kafkaSplitting` setting in the [mirrord-operator Helm chart](https://github.com/metalbear-co/charts/blob/main/mirrord-operator/values.yaml). {% endstep %} {% step %} -#### Configure the operator's Kafka client +### Configure the operator's Kafka client The mirrord operator will need to be able to perform some operations on the Kafka cluster. To allow for properly configuring the operator's Kafka client, on operator installation with `operator.kafkaSplitting` enabled, @@ -433,7 +433,7 @@ See [additional options](queue-splitting.md#additional-options) section for more {% endstep %} {% step %} -#### Authorize deployed consumers +### Authorize deployed consumers In order to be targeted with Kafka splitting, a deployed consumer must be able to use the temporary queues created by mirrord. E.g. if the consumer application describes the queue or reads messages from it — it must be able to do the same on a temporary queue. @@ -442,7 +442,7 @@ This might require extra actions on your side to adjust the authorization, for e {% endstep %} {% step %} -#### Provide application context +### Provide application context On operator installation with `operator.kafkaSplitting` enabled, a new [`CustomResource`](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) type is defined in your cluster @@ -484,7 +484,7 @@ If the variable is absent, the fallback value `views-topic` is used instead. The Kafka consumer group id is read from environment variable `KAFKA_GROUP_ID` in container `consumer`. 3. The Kafka queue can be referenced in a mirrord config under ID `views-topic`. -##### Link the topics consumer resource to the deployed consumer +#### Link the topics consumer resource to the deployed consumer The topics consumer resource is namespaced, so it can only reference a Kafka consumer deployed in the same namespace. The reference is specified with `spec.consumer*` fields, which cover api version, kind, and name of the Kubernetes workload. @@ -498,7 +498,7 @@ consumerName: kafka-notifications-worker The operator supports Kafka splitting on deployments, stateful sets, and Argo rollouts. -##### Desribe consumed queues in the topics consumer resource +#### Desribe consumed queues in the topics consumer resource The topics consumer resource describes Kafka queues consumed by the referenced consumer. The queues are described in entries of the `spec.topics` list: @@ -520,9 +520,9 @@ The mirrord operator can only read consumer's environment variables if they are {% endstep %} {% endstepper %} -#### Additional Options +### Additional Options -##### Customizing Temporary Kafka Queue Names +#### Customizing Temporary Kafka Queue Names {% hint style="info" %} Available since chart version `1.27` and operator version `3.114.0`. @@ -547,7 +547,7 @@ The provided format must contain the three variables: `{{RANDOM}}`, `{{FALLBACK} * `{{FALLBACK}}` will resolve either to `-fallback-` or `-` literal. * `{{ORIGINAL_TOPIC}}` will resolve to the name of the original topic that is being split. -##### Reusing Kafka Client Configs +#### Reusing Kafka Client Configs `MirrordKafkaClientConfig` resource supports property inheritance via `spec.parent` field. When resolving a resource `config-A` that has a parent `config-B`: @@ -594,7 +594,7 @@ bootstrap.servers=kafka.default.svc.cluster.local:9092 client.id=mirrord-operator ``` -##### Configuring Kafka Clients with Secrets +#### Configuring Kafka Clients with Secrets `MirrordKafkaClientConfig` also supports loading properties from a Kubernetes [`Secret`](https://kubernetes.io/docs/concepts/configuration/secret/), with the `spec.loadFromSecret` field. The value for `spec.loadFromSecret` is given in the form: `/`. @@ -626,7 +626,7 @@ spec: Note that by default, mirrord operator has read access only to the secrets in the operator's namespace. {% endhint %} -##### Configuring Custom Kafka Authentication +#### Configuring Custom Kafka Authentication For authentication methods that cannot be handled just by setting [client properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md), we provide a separate field `spec.authenticationExtra`. The field allows for specifying custom authentication methods: @@ -661,7 +661,7 @@ Note that operator's service account can be annotated with the IAM role's ARN wi {% endtab %} {% endtabs %} -##### Configuring Workload Restart +#### Configuring Workload Restart To inject the names of the temporary queues into the consumer workload, the operator always requires the workload to be restarted. @@ -683,14 +683,14 @@ is started before the TTL elapses. Specified in seconds. {% stepper %} {% step %} -#### Enable RabbitMQ splitting in the Helm chart +### Enable RabbitMQ splitting in the Helm chart Enable the `operator.rmqSplitting` setting in the [mirrord-operator Helm chart](https://github.com/metalbear-co/charts/blob/main/mirrord-operator/values.yaml). {% endstep %} {% step %} -#### Cluster Declaration +### Cluster Declaration The mirrord operator needs a way to connect to your RabbitMQ cluster to consume and re-route messages according to filters. As part of operator installation with `operator.rmqSplitting` enabled, a new [`CustomResource`](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) type is defined in your cluster — `MirrordPropertyList`. Use this resource to define the cluster and queue connection parameters for splitting. A `MirrordPropertyList` must live in the same namespace as the consumer workload (and the `MirrordWorkloadQueueRegistry`), which may very well be different than your RabbitMQ broker's namespace. @@ -755,7 +755,7 @@ spec: {% endhint %} -##### Cluster Properties +#### Cluster Properties | Property | Description | Required | Type | Default | | --------------------- | :-----------------------------------------------------------------: | :------: | :------------------------------------------------------------:|:----------------------------------:| @@ -771,7 +771,7 @@ spec: | `ca-certificates.crt` | CA certificate(s) (PEM format) used to verify the broker's identity | | string (PEM) | | | `client.*` | Custom metadata or properties sent to the broker | | object / key-value pairs | | -##### Queue Declare Properties +#### Queue Declare Properties | Property | Description | Required | Type | Default | | --------------------- | :---------------------------------------------------------------------------------------------------------------: | :------: | :-----------------------:|:---------:| @@ -783,7 +783,7 @@ spec: {% endstep %} {% step %} -#### Provide application context +### Provide application context On operator installation with `operator.rmqSplitting` enabled, a new [`CustomResource`](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) type is defined in your cluster — `MirrordWorkloadQueueRegistry`. Users with permissions to get CRDs can verify its existence @@ -824,7 +824,7 @@ The registry above says that: 3. The container consumes two RabbitMQ queues. Their names are read from environment variables `INCOMING_MEME_QUEUE_NAME` and `AD_QUEUE_NAME`. 4. The queues can be referenced in a mirrord config under IDs `meme-queue` and `ad-queue`, respectively. -##### Link the registry to the deployed consumer +#### Link the registry to the deployed consumer The queue registry is a namespaced resource, so it can only reference a consumer deployed in the same namespace. The reference is specified with `spec.consumer`: @@ -832,7 +832,7 @@ The reference is specified with `spec.consumer`: * `workloadType` — type of the Kubernetes workload of the deployed consumer. Right now only consumers deployed in deployments and rollouts are supported. * `container` — name of the exact container running in the workload. This field is optional. If you omit it, the registry will reference all of the workload's containers. -##### Describe consumed queues in the registry +#### Describe consumed queues in the registry The queue registry describes RabbitMQ queues consumed by the referenced consumer. The queues are described in entries of the `spec.queues` object. @@ -867,14 +867,14 @@ The mirrord operator can only read consumer's environment variables if they are {% stepper %} {% step %} -#### Enable GCP Pub/Sub splitting in the Helm chart +### Enable GCP Pub/Sub splitting in the Helm chart Enable the `operator.gcpPubsubSplitting` setting in the [mirrord-operator Helm chart](https://github.com/metalbear-co/charts/blob/main/mirrord-operator/values.yaml). {% endstep %} {% step %} -#### Authenticate the mirrord operator +### Authenticate the mirrord operator The mirrord operator needs access to the Google Cloud Pub/Sub API to create and manage temporary topics and subscriptions. @@ -957,14 +957,14 @@ A good starting point is to assign the `roles/pubsub.editor` role to the operato {% endstep %} {% step %} -#### Authorize deployed consumers +### Authorize deployed consumers In order to be targeted with Pub/Sub splitting, a deployed consumer must be able to read from the temporary subscriptions created by mirrord. If the consumer's IAM permissions are scoped to specific subscription names, you will need to extend them to cover subscriptions with the `mirrord-tmp-` prefix. This prefix is customizable via the `spec.tmpNameTemplate` field in your `MirrordSplitConfig` resource. {% endstep %} {% step %} -#### Provide application context +### Provide application context On operator installation with `operator.gcpPubsubSplitting` enabled, a new [`CustomResource`](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) type is defined in your cluster - `MirrordSplitConfig`. Users with permissions to get CRDs can verify its existence with `kubectl get crd mirrordsplitconfigs.queues.mirrord.metalbear.co`. @@ -1007,14 +1007,14 @@ The `MirrordSplitConfig` above says that: 4. The GCP project ID is in environment variable `GCP_PROJECT_ID` in container `consumer`. 5. The subscription can be referenced in a mirrord config under ID `user-events`. -##### Link the config to the deployed consumer +#### Link the config to the deployed consumer The `MirrordSplitConfig` is a namespaced resource. The target workload reference is specified with `spec.targetRef`: * `apiVersion` - API version of the Kubernetes workload (e.g. `apps/v1`). * `kind` - type of the workload. Supported: `Deployment`, `StatefulSet`, `Rollout`. * `name` - name of the workload. -##### Describe consumed subscriptions +#### Describe consumed subscriptions Each entry in the `spec.queues` list describes one or more Pub/Sub subscriptions consumed by the workload: @@ -1087,14 +1087,14 @@ The mirrord operator can only read consumer's environment variables if they are {% stepper %} {% step %} -#### Enable Azure Service Bus splitting in the Helm chart +### Enable Azure Service Bus splitting in the Helm chart Enable the `operator.azureServiceBusSplitting` setting in the [mirrord-operator Helm chart](https://github.com/metalbear-co/charts/blob/main/mirrord-operator/values.yaml). {% endstep %} {% step %} -#### Authenticate the mirrord operator +### Authenticate the mirrord operator The mirrord operator needs to connect to your Azure Service Bus namespace to consume and re-route messages. You have three options for authentication: @@ -1113,7 +1113,7 @@ Register an Azure AD application, create a client secret, and assign it the **Az {% endstep %} {% step %} -#### Create a MirrordPropertyList +### Create a MirrordPropertyList As part of operator installation with `operator.azureServiceBusSplitting` enabled, the `MirrordPropertyList` custom resource type is available in your cluster. Create one with your Service Bus connection details. @@ -1207,7 +1207,7 @@ spec: {% endtabs %} -##### Property Reference +#### Property Reference | Property | Description | Required | | -------- | :---------: | :------: | @@ -1220,7 +1220,7 @@ spec: {% endstep %} {% step %} -#### Create a MirrordSplitConfig +### Create a MirrordSplitConfig Create a `MirrordSplitConfig` resource for the target workload. Azure Service Bus uses `kind: azureServiceBus` in queue entries and supports both the Queue model and the Topic/Subscription model. @@ -1274,7 +1274,7 @@ spec: The `clientConfigs.azureServiceBus` field points to the `MirrordPropertyList` you created in the previous step. You can override it per-queue using the `clientConfig` field on individual queue entries. -##### AppConfig reference fields +#### AppConfig reference fields Each item in `queue`, `topic`, or `subscription` is an `AppConfigRef` that describes how to find the resource name in the workload's environment: @@ -1303,7 +1303,7 @@ queues: - envLike: "^SB_QUEUE_.*" ``` -##### Preserving the value format +#### Preserving the value format By default the operator treats the whole environment variable value as the resource name and replaces it with a temporary one. When the application reads the name as part of a larger string - a URL, a resource path, or a connection string - replacing the whole value would break it. You can use`valuePattern` to solve this: it is a regex whose capture group marks the part of the value that is the resource name. The operator swaps only that captured part for the temporary name and keeps everything around it unchanged. @@ -1323,7 +1323,7 @@ queues: With `PUBSUB_SUBSCRIPTION=gcppubsub://projects/my-project/subscriptions/orders`, the operator captures `orders`, creates a temporary subscription, and rewrites the variable to `gcppubsub://projects/my-project/subscriptions/`, so the application still gets a full URL. -##### Per-queue client configuration +#### Per-queue client configuration To use a different `MirrordPropertyList` for a specific queue entry (instead of the default from `clientConfigs.azureServiceBus`), set the `clientConfig` field: @@ -1337,7 +1337,7 @@ queues: - env: SERVICE_BUS_QUEUE_NAME ``` -##### Wildcard queue ID +#### Wildcard queue ID You can use `*` as a queue ID in the mirrord config to apply a filter to all queues defined in the `MirrordSplitConfig`: @@ -1367,11 +1367,11 @@ The mirrord operator can only read the consumer's environment variables if they {% endstep %} {% step %} -#### Additional options +### Additional options The `MirrordSplitConfig` supports several optional fields that control restart behavior, temporary resource naming, and drain timing. -##### Restart policy +#### Restart policy Controls how the workload is restarted when patched for queue splitting: @@ -1389,7 +1389,7 @@ spec: | `timeout` | Seconds to wait for pods to become ready after restart | 60 | | `waitForPods` | Number of patched pods required before sessions may start, or `"all"` | 1 | -##### Drain timeout +#### Drain timeout After all splitting sessions end, the operator waits for the fallback queue to drain before deleting temporary resources. Set `drainTimeout` (in seconds) to cap how long this wait lasts: @@ -1400,7 +1400,7 @@ spec: Set to `0` to skip draining entirely (temporary queues are deleted immediately). If omitted, the default is 30 seconds. -##### Temporary resource name template +#### Temporary resource name template You can customize the naming format of temporary queues/topics created by the operator: @@ -1421,7 +1421,7 @@ Azure Service Bus resource names can be up to 260 characters. If the rendered na {% endtab %} {% endtabs %} -### Setting a Filter for a mirrord Run +## Setting a Filter for a mirrord Run Once cluster setup is done, mirrord users can start running sessions with queue message filters in their mirrord configuration files. [`feature.split_queues`](https://metalbear.com/mirrord/docs/config/options#feature-split_queues) is the configuration field they need to specify in order to filter queue messages. @@ -1708,9 +1708,9 @@ Both `message_filter` and `jq_filter` can be combined - a message must match bot {% endtab %} {% endtabs %} -### FAQ +## FAQ -##### How do I authenticate operator's Kafka client with an SSL certificate? +#### How do I authenticate operator's Kafka client with an SSL certificate? An example `MirrordKafkaClientConfig` would look as follows: @@ -1762,7 +1762,7 @@ spec: properties: [] ``` -##### How do I authenticate operator's Kafka client with a Java KeyStore? +#### How do I authenticate operator's Kafka client with a Java KeyStore? The mirrord operator does not support direct use of JKS files. In order to use JKS files with Kafka splitting, first extract all necessary certificates and key to PEM files. @@ -1795,7 +1795,7 @@ openssl pkcs12 -in truststore.p12 -nokeys -out ca-cert.pem Then, follow the guide for [authenticating with an SSL certificate](queue-splitting.md#how-do-i-authenticate-operators-kafka-client-with-an-ssl-certificate). -### Troubleshooting SQS splitting +## Troubleshooting SQS splitting If you're trying to use SQS-splitting and are facing difficulties, here are some steps you can go through to identify and hopefully solve the problem. @@ -1839,14 +1839,14 @@ First, some generally applicable steps: kubectl rollout restart deployment mirrord-operator -n mirrord ``` -##### If some (but not all) of the messages that should arrive at the local service arrive at the remote service +#### If some (but not all) of the messages that should arrive at the local service arrive at the remote service It's possible the target workload's restart is not complete yet, and there are still pods reading directly from the original queue (those will be pods that DO NOT have a `operator.metalbear.co/patched` label). You can wait a bit for them to be replaced with new pods, patched by mirrord, that read from a temporary queue created by mirrord, or you can delete them. -##### If all SQS sessions are over but the remote service still didn't change back to read from the original queue +#### If all SQS sessions are over but the remote service still didn't change back to read from the original queue When there are no more queue splitting sessions to a target, the target workload will not immediately be changed to read directly from the original queue. Instead, it will keep reading from the temporary queue until its empty, so that no @@ -1864,7 +1864,7 @@ If that service is trying to consume messages correctly, and the temporary queue application still doesn't get restored to its original state, please try restarting the application, deleting any lingering `MirrordSqsSession` objects, and if possible, restart the mirrord operator. -### Troubleshooting Azure Service Bus splitting +## Troubleshooting Azure Service Bus splitting If you are having issues with Azure Service Bus splitting, start with these general steps: @@ -1886,19 +1886,19 @@ If you are having issues with Azure Service Bus splitting, start with these gene helm upgrade mirrord-operator --reuse-values --set operator.logLevel "mirrord=info,operator=info,operator_queue_splitting::azure_service_bus=trace" metalbear/mirrord-operator ``` -##### Messages are not reaching the local application +#### Messages are not reaching the local application Check that: - The producer is setting AMQP application properties on messages. The operator routes based on these properties when `message_filter` is used. - Your `message_filter` regex patterns match the actual property values. Property values are compared as plain strings. - If using `jq_filter`, verify the message body is valid JSON and the jq expression returns `true` for your test messages. -##### Authentication errors in operator logs +#### Authentication errors in operator logs - **Connection string auth**: verify the connection string is correct and the SAS key has Manage, Send, and Listen claims. - **Workload Identity / Managed Identity**: verify the managed identity has the **Azure Service Bus Data Owner** role on the namespace. Check that AKS Workload Identity is properly configured and the operator's service account has the correct annotations. - **Service Principal**: verify that `tenant_id`, `client_id`, and `client_secret` are all present in the `MirrordPropertyList` and that the app registration has the correct role assignment. -##### Temporary queues are not being cleaned up +#### Temporary queues are not being cleaned up After all splitting sessions end, the operator deletes temporary queues. If they linger, check that the operator has `Manage` rights on the Service Bus namespace and that the operator pod is running. You can also set `drainTimeout` in the `MirrordSplitConfig` to control how long fallback queues are kept alive after sessions end. diff --git a/docs/troubleshooting/utilities.md b/docs/troubleshooting/utilities.md index 359d331d..16a13f71 100644 --- a/docs/troubleshooting/utilities.md +++ b/docs/troubleshooting/utilities.md @@ -4,21 +4,21 @@ date: 2026-02-02T12:59:39.000Z description: Lightweight options that provide additional insight when debugging mirrord --- -## Response Header Injection +# Response Header Injection Indicate whether the request was handled by mirrord. {% hint style="info" %} This feature requires at least mirrord-agent version **3.163.0**. {% endhint %} -### Use Case +## Use Case When debugging, it’s often useful to know how a response was handled by mirrord after being intercepted. This option helps verify mirrord’s routing decisions and understand whether the intercepted traffic was forwarded to the local process or passed through to its original destination. -### Header behavior +## Header behavior When enabled, the mirrord agent automatically adds a `mirrord-agent` header to HTTP responses handled by mirrord. -#### Possible values for the header: +### Possible values for the header: - `forwarded-to-client`: The mirrord agent intercepted the request, sent it to the local process, and then passed back the response (mirrord handled it). - `passed-through`: The mirrord agent intercepted the request, sent it to its original destination, and then passed back the response (the request was not handled by the local process). For example, if a request doesn’t match active filters, it will be passed to the original destination. @@ -35,10 +35,10 @@ You can enable it with the following configuration in the `mirrord.json` file: You can see all the agent configuration options [here](https://metalbear.com/mirrord/docs/config#agent). -## Latency Diagnose +# Latency Diagnose The `mirrord diagnose latency` command helps identify network latency issues between your local environment and the target workload. It measures round-trip time (RTT) across multiple iterations and reports latency statistics, making it useful when mirrord feels slow or requests take longer than expected. -### When to use this +## When to use this Use this command if you experience: - Slow request/response times when using mirrord @@ -54,7 +54,7 @@ Options `-h, --help`: Print help information. -#### Example +### Example ```bash mirrord diagnose latency ``` diff --git a/docs/use-cases/mirrord-for-ci.md b/docs/use-cases/mirrord-for-ci.md index e3649c88..2c02c311 100644 --- a/docs/use-cases/mirrord-for-ci.md +++ b/docs/use-cases/mirrord-for-ci.md @@ -29,12 +29,12 @@ it as a background process. The `mirrord ci start` command is more appropriate for this use case, since it starts your app and mirrord as background processes, allowing you to then run tests while your app is running in the background and connected to the cluster. -### Prerequisites +## Prerequisites 1. Minimum mirrord CLI version `3.181.0`. 2. The CI runner must be able to access the Kubernetes cluster in which you want to test. -### Kubernetes requirements +## Kubernetes requirements The CI runner must be able to access the Kubernetes cluster where the service you want to target is deployed, otherwise mirrord won't work. You'll need a @@ -45,7 +45,7 @@ CI runner that points to the target's cluster, and has the appropriate It's recommended that you create a Kubernetes [service account](https://kubernetes.io/docs/concepts/security/service-accounts/) for the CI runner. -### For mirrord for Teams users +## For mirrord for Teams users {% hint style="info" %} This feature is available to users on the Enterprise pricing plan. @@ -62,7 +62,7 @@ mirrord ci api-key Copy it and save it as the **secret** environment variable `MIRRORD_CI_API_KEY` in your CI. -### Starting a mirrord CI session +## Starting a mirrord CI session The `mirrord ci start` command is used to start the service being tested in your CI runner, and supports the same arguments as `mirrord exec`, including specifying a target with `--target` or using a configuration file with `--config-file`. Here’s an example for starting a Go service called `ip-visit-counter`: @@ -81,7 +81,7 @@ You can start multiple mirrord for CI sessions during a single CI job by running If you want to run the service with mirrord in the foreground, you can use the `--foreground` arg. {% endhint %} -### Starting a mirrord CI session in a container +## Starting a mirrord CI session in a container Use `mirrord ci container` when your CI job runs inside a local container runtime such as Docker. ```sh @@ -89,7 +89,7 @@ mirrord ci container -- docker run my-image ``` -#### Application logs +### Application logs By default, `stdout` and `stderr` outputs from your application are saved to a file in the OS' temporary directory (e.g. `/tmp/mirrord`). A directory is created based on the name @@ -125,7 +125,7 @@ When running with the --foreground argument, application logs are streamed to th process’s `stdout` and `stderr` unless `ci.output_dir` is configured. {% endhint %} -### Stopping a mirrord CI session +## Stopping a mirrord CI session After the tests are done, you should stop the mirrord CI session using `mirrord ci stop`. It's recommended that you do it, even if you won't be running mirrord for another service in this CI runner. diff --git a/docs/using-mirrord/multi-cluster-setup.md b/docs/using-mirrord/multi-cluster-setup.md index 90767937..a03a13bd 100644 --- a/docs/using-mirrord/multi-cluster-setup.md +++ b/docs/using-mirrord/multi-cluster-setup.md @@ -5,7 +5,7 @@ description: "Step-by-step guide to setting up multi-cluster mirrord: install th This guide covers how to set up multi-cluster mirrord. It involves installing the operator on all clusters, choosing an authentication method, and configuring the Primary cluster to connect to downstream clusters. -### Prerequisites +## Prerequisites Before you start, make sure you have: @@ -16,17 +16,17 @@ Before you start, make sure you have: --- -### Authentication Methods +## Authentication Methods For each downstream cluster, you must specify an `authType` that determines how the Primary mirrord operator authenticates to it. -#### Bearer Token (`authType: bearerToken`) +### Bearer Token (`authType: bearerToken`) Uses ServiceAccount tokens that are automatically refreshed via the Kubernetes TokenRequest API. Good for most setups where the Primary cluster can reach the downstream cluster's API server. You generate an initial token manually during setup. After that, the operator auto-refreshes the token before it expires using the TokenRequest API. The refreshed token keeps the same lifetime as the original. -#### EKS IAM (`authType: eks`) +### EKS IAM (`authType: eks`) For AWS EKS clusters. The Primary operator generates short-lived tokens using its IAM role (via IRSA). No secrets to manage - tokens are generated and refreshed automatically every 10 minutes. @@ -34,7 +34,7 @@ On the Primary cluster, the operator pod gets AWS credentials through IRSA (`sa. No Kubernetes Secret is needed - authentication is entirely through IAM. -#### AKS Workload Identity (`authType: aks`) +### AKS Workload Identity (`authType: aks`) For Azure AKS clusters. The Primary operator generates tokens by exchanging its projected ServiceAccount token with Azure AD. No secrets to manage - tokens are generated and refreshed automatically at the halfway point of the token's lifetime (~12 hours for a typical 24-hour Azure AD token). @@ -42,13 +42,13 @@ On the Primary cluster, the operator pod gets Azure credentials through [Workloa No Kubernetes Secret is needed - authentication is entirely through Azure AD. -#### mTLS (`authType: mtls`) +### mTLS (`authType: mtls`) For clusters that require client certificate authentication. You provide the client certificate and key in the cluster configuration or Secret. Kubernetes does not auto-refresh mTLS client certificates. You are responsible for rotating the certificates you provide before they expire. -#### Fields per Auth Type +### Fields per Auth Type | Field | `bearerToken` | `eks` | `aks` | `mtls` | |-------|:---:|:---:|:---:|:---:| @@ -63,16 +63,16 @@ Kubernetes does not auto-refresh mTLS client certificates. You are responsible f --- -### Setting Up Downstream Clusters +## Setting Up Downstream Clusters Every downstream cluster needs the mirrord operator installed with the `operator.multiClusterMember` helm chart value set to `true`. This creates the `ServiceAccount`, `ClusterRoles`, and `ClusterRoleBindings` that the Primary operator needs to manage sessions on that cluster. -#### Bearer Token / mTLS Clusters +### Bearer Token / mTLS Clusters {% stepper %} {% step %} -#### Install the operator on the downstream cluster +### Install the operator on the downstream cluster ```bash helm install mirrord-operator metalbear/mirrord-operator \ @@ -84,7 +84,7 @@ This creates a `mirrord-operator-envoy` ServiceAccount with the necessary RBAC p {% endstep %} {% step %} -#### Generate an initial token (bearer token auth only) +### Generate an initial token (bearer token auth only) ```bash kubectl create token mirrord-operator-envoy -n mirrord --duration=24h @@ -97,11 +97,11 @@ For mTLS, skip this step. Instead, you'll provide the client certificate and key {% endstep %} {% endstepper %} -#### EKS IAM Clusters +### EKS IAM Clusters EKS IAM authentication lets the Primary operator authenticate to downstream EKS clusters using its AWS IAM role. No Kubernetes Secrets to manage — the operator generates short-lived tokens from its IAM identity. -##### How EKS IAM Authentication Works +#### How EKS IAM Authentication Works The Primary operator pod needs to talk to downstream EKS clusters. To do that, it needs a token. Here's how the token gets created and accepted: @@ -115,7 +115,7 @@ The Primary operator pod needs to talk to downstream EKS clusters. To do that, i 5. **Kubernetes RBAC grants permissions** — the ClusterRoleBindings on the downstream cluster (created by Helm with `multiClusterMemberIamGroup`) grant the `mirrord-operator-envoy` group the necessary permissions. -##### What Goes Where +#### What Goes Where | Component | Where | Purpose | |-----------|-------|---------| @@ -129,12 +129,12 @@ The Primary operator pod needs to talk to downstream EKS clusters. To do that, i The Primary cluster does **not** need an Access Entry. The operator pod runs inside the Primary cluster, so it authenticates using its ServiceAccount — no IAM token needed. The Access Entries are only needed on downstream clusters where the pod authenticates from the outside. {% endhint %} -##### Setup Steps +#### Setup Steps {% stepper %} {% step %} -#### Associate an OIDC Identity Provider with the Primary cluster +### Associate an OIDC Identity Provider with the Primary cluster This is a one-time AWS setup. It registers the Primary EKS cluster's OIDC issuer as an IAM Identity Provider, which is what enables IRSA — the mechanism that lets pods assume IAM roles. @@ -148,7 +148,7 @@ Skip this if already done (e.g. if other workloads in the cluster already use IR {% endstep %} {% step %} -#### Create the IAM role with a trust policy +### Create the IAM role with a trust policy Create an IAM role that the operator pod will assume. The trust policy controls who can assume this role — it should only allow the Primary operator's ServiceAccount. @@ -188,7 +188,7 @@ This IAM role does **not** need any IAM policies attached (no `s3:*`, `sqs:*`, e {% endstep %} {% step %} -#### Create an EKS Access Entry on each downstream cluster +### Create an EKS Access Entry on each downstream cluster This is an AWS-level configuration (not a Kubernetes resource). It tells each downstream EKS cluster: "When this IAM role authenticates, map it to the `mirrord-operator-envoy` Kubernetes group." @@ -207,7 +207,7 @@ The Access Entry itself doesn't grant any Kubernetes permissions — it only est {% endstep %} {% step %} -#### Install the operator on each downstream cluster +### Install the operator on each downstream cluster Install the operator with `multiClusterMember` and `multiClusterMemberIamGroup`: @@ -222,7 +222,7 @@ This creates ClusterRoleBindings that grant the `mirrord-operator-envoy` Kuberne {% endstep %} {% step %} -#### Install the operator on the Primary cluster with `sa.roleArn` +### Install the operator on the Primary cluster with `sa.roleArn` On the Primary cluster, set `sa.roleArn` so the operator pod can assume the IAM role: @@ -238,11 +238,11 @@ See the [Configuring the Primary Cluster](#configuring-the-primary-cluster) sect {% endstep %} {% endstepper %} -#### AKS Workload Identity Clusters +### AKS Workload Identity Clusters AKS Workload Identity lets the Primary operator authenticate to downstream AKS clusters using its Azure Managed Identity. No Kubernetes Secrets to manage - the operator exchanges its projected SA token with Azure AD for access tokens. -##### How AKS Workload Identity Authentication Works +#### How AKS Workload Identity Authentication Works The Primary operator pod needs to talk to downstream AKS clusters. To do that, it needs a token. Here's how the token gets created and accepted: @@ -256,7 +256,7 @@ The Primary operator pod needs to talk to downstream AKS clusters. To do that, i 5. **Kubernetes RBAC grants permissions** - the ClusterRoleBindings on the downstream cluster (created by Helm with `multiClusterMemberAzureGroup`) grant the Azure AD group the necessary permissions. -##### What Goes Where +#### What Goes Where | Component | Where | Purpose | |-----------|-------|---------| @@ -271,12 +271,12 @@ The Primary operator pod needs to talk to downstream AKS clusters. To do that, i The Primary cluster does **not** need special RBAC for itself. The operator pod runs inside the Primary cluster, so it authenticates using its ServiceAccount - no Azure AD token needed. The Federated Identity Credential and role assignments are only needed for downstream clusters where the pod authenticates from the outside. {% endhint %} -##### Setup Steps +#### Setup Steps {% stepper %} {% step %} -#### Enable Workload Identity on the Primary AKS cluster +### Enable Workload Identity on the Primary AKS cluster If not already enabled: @@ -291,7 +291,7 @@ az aks update \ {% endstep %} {% step %} -#### Create a User-Assigned Managed Identity +### Create a User-Assigned Managed Identity ```bash az identity create \ @@ -305,7 +305,7 @@ Note the `clientId` from the output - you'll need it for `sa.azureClientId`. {% endstep %} {% step %} -#### Create a Federated Identity Credential +### Create a Federated Identity Credential This tells Azure AD: "when a token comes from the Primary cluster's OIDC issuer, signed for the `mirrord-operator` ServiceAccount, trust it as this Managed Identity." @@ -333,7 +333,7 @@ The `--subject` must match the operator's ServiceAccount namespace and name exac {% endstep %} {% step %} -#### Enable Azure AD on each downstream AKS cluster +### Enable Azure AD on each downstream AKS cluster Each downstream AKS cluster must have Azure AD integration enabled so it can validate Azure AD tokens from the Managed Identity: @@ -351,7 +351,7 @@ If you skip this step, the downstream cluster's API server won't understand Azur {% endstep %} {% step %} -#### Grant access on each downstream AKS cluster +### Grant access on each downstream AKS cluster The Managed Identity needs Kubernetes-level permissions on each downstream cluster to manage sessions. Bind the Managed Identity's `principalId` (object ID) directly to the ClusterRoles created by Helm: @@ -377,7 +377,7 @@ The `mirrord-operator-envoy` and `mirrord-operator-envoy-remote` ClusterRoles ar {% endstep %} {% step %} -#### Install the operator on each downstream cluster +### Install the operator on each downstream cluster Install the operator with `multiClusterMember` and `multiClusterMemberAzureGroup`: @@ -392,7 +392,7 @@ This creates ClusterRoleBindings that grant the `mirrord-operator-envoy` group t {% endstep %} {% step %} -#### Install the operator on the Primary cluster with `sa.azureClientId` +### Install the operator on the Primary cluster with `sa.azureClientId` On the Primary cluster, set `sa.azureClientId` so the Workload Identity webhook injects Azure credentials into the pod. You also need the downstream cluster's CA certificate (`caData`) since AKS clusters use per-cluster self-signed CAs: @@ -419,11 +419,11 @@ See the [Configuring the Primary Cluster](#configuring-the-primary-cluster) sect --- -### Configuring the Primary Cluster +## Configuring the Primary Cluster Install the operator on the Primary cluster with multi-cluster enabled and all downstream clusters configured. -#### Helm Values +### Helm Values ```yaml operator: @@ -488,13 +488,13 @@ sa: The cluster key names in the `clusters` map should match the real cluster names. For EKS clusters this is especially important — the operator uses the key as the EKS cluster name when signing IAM tokens. {% endhint %} -#### Where Data Is Stored +### Where Data Is Stored When you provide cluster configuration in the Helm values, the chart splits it into two places. Non-sensitive configuration (`server`, `caData`, `authType`, `region`, `isDefault`, `namespace`) goes into the ConfigMap (`clusters-config.yaml`). Sensitive credentials (`bearerToken`, `tls.crt`, `tls.key`) go into a Secret (`mirrord-cluster-`). For EKS IAM and AKS Workload Identity clusters, no Secret is created - everything is in the ConfigMap since authentication is through IAM/Azure AD, not stored credentials. -#### Manual Secret Creation +### Manual Secret Creation If you prefer to manage secrets outside of Helm values, you can create the Secret manually. The Secret must be labeled with `operator.metalbear.co/remote-cluster-credentials=true` and named `mirrord-cluster-`. The cluster configuration (server, authType, etc.) still needs to be in the Helm values or the `clusters-config.yaml` ConfigMap. @@ -535,7 +535,7 @@ EKS IAM and AKS Workload Identity clusters do not need a Secret at all. They aut --- -### RBAC — How Permissions Work +## RBAC — How Permissions Work When the Primary operator connects to a downstream cluster, it needs permissions to list targets, create sessions, run health checks, and more. These permissions are set up automatically by the Helm chart on each downstream cluster. @@ -550,7 +550,7 @@ These multi-cluster roles are separate from the Operator API roles used by mirro A ClusterRole by itself doesn't grant anything — it only defines what actions are possible. ClusterRoleBindings connect the ClusterRole to an identity (a ServiceAccount or a group). -#### How Bindings Differ by Auth Type +### How Bindings Differ by Auth Type | Auth type | Identity bound to ClusterRoles | How it's set up | |-----------|-------------------------------|----------------| @@ -570,7 +570,7 @@ In practice: --- -### Verify the Connection +## Verify the Connection After installing the operator on all clusters, verify that the Primary can reach all downstream clusters: @@ -599,7 +599,7 @@ Each connected cluster should show `license_fingerprint` and `operator_version`. --- -### Token Refresh +## Token Refresh | Auth type | Refresh mechanism | Notes | |-----------|------------------|-------| @@ -610,7 +610,7 @@ Each connected cluster should show `license_fingerprint` and `operator_version`. --- -### FAQ +## FAQ **Q: Do developers need to know about multi-cluster?** A: No. The developer experience is identical to single-cluster. Developers run `mirrord exec` as usual and the operator handles everything. Note that multi-cluster sessions only work when the developer connects to the Primary cluster — connecting directly to a downstream cluster will start a regular single-cluster session on that cluster. diff --git a/docs/using-mirrord/multiple-concurrent-sessions.md b/docs/using-mirrord/multiple-concurrent-sessions.md index 86ee6234..20e024f5 100644 --- a/docs/using-mirrord/multiple-concurrent-sessions.md +++ b/docs/using-mirrord/multiple-concurrent-sessions.md @@ -11,10 +11,10 @@ weight: 140 toc: true tags: ["open source", "team", "enterprise"] --- -## Introduction +# Introduction `mirrord up` allows creating and running multiple mirrord sessions based on configuration defined in a single file — think `docker compose` but for mirrord. This can be useful for cases when you need to debug multiple related microservices and would like to manage their lifecycle together. -## Getting started +# Getting started The fastest way to get a valid `mirrord-up.yaml` is the interactive wizard: ```sh @@ -49,11 +49,11 @@ This will start all defined services, and they will run in parallel. The `mirror Services default to `split` mode, which steals incoming traffic matching an `http_filter`. When no filter is provided, mirrord generates one based on the session key: `baggage: .*mirrord-session={key}.*`. -## Configuration +# Configuration -### Config file (`mirrord-up.yaml`) +## Config file (`mirrord-up.yaml`) -#### `common` +### `common` Common configuration options, applied to all defined services. Currently 3 options are supported: - [`accept_invalid_certificates`](https://metalbear.com/mirrord/docs/config/options#root-accept_invalid_certificates) - [`operator`](https://metalbear.com/mirrord/docs/config/options#root-operator) @@ -61,10 +61,10 @@ Common configuration options, applied to all defined services. Currently 3 optio All 3 map directly to their `mirrord.json` counterparts. -#### `services` +### `services` A map from service ids to a `ServiceConfig`. Each entry in this map defines and configures a mirrord process that will be run as part of the session. -##### `services.*.target` +#### `services.*.target` Specifies the target of the session. Has 2 fields: `path` and `namespace`, which map directly to their `mirrord.json` counterparts. Examples: ```yaml @@ -86,23 +86,23 @@ target: path: deployment/test-app ``` -##### `services.*.env` +#### `services.*.env` Specifies the environment variable configuration for the given service. Maps directly (1:1) to [`feature.env`](https://metalbear.com/mirrord/docs/config/options#feature-env) -##### `services.*.default_mode` +#### `services.*.default_mode` So far, only `split` is supported. The incoming mode is set to `steal` with http filter. User-provided filter is used if provided, otherwise defaulting to `baggage: .*mirrord-session={key}.*`. -##### `services.*.http_filter` +#### `services.*.http_filter` Specifies the HTTP filtering configuration for the given service. Maps directly to [`feature.network.incoming.http_filter`](https://metalbear.com/mirrord/docs/config/options#feature-network-incoming) -##### `services.*.ignore_ports` +#### `services.*.ignore_ports` List of ports that should be ignored in incoming traffic. Maps directly to [`feature.network.incoming.ignore_ports`](https://metalbear.com/mirrord/docs/config/options#feature-network-incoming) -##### `services.*.messages` +#### `services.*.messages` Specifies queue splitting configuration (Not supported as of now). -##### `services.*.run` +#### `services.*.run` Specifies the command that should be run with mirrord. Has 2 fields: - `command`: Array of strings containing the command to be run and its CLI arguments. - `type`: can be either `exec` or `container`, defaults to `exec`. Specifies how mirrord should be run (i.e. with `mirrord exec` or `mirrord container`) @@ -121,15 +121,15 @@ run: ``` -## CLI args +# CLI args -### `-f`, `--config-file` +## `-f`, `--config-file` Allows specifying a different config file, e.g. `mirrord up -f mirrord-up-custom.yaml` -### `--key` +## `--key` Allows specifying a custom session key. When not supplied, the OS username is used. -## `mirrord up init` +# `mirrord up init` Interactive wizard that generates a skeleton `mirrord-up.yaml`. Targets are entered as freeform strings (e.g. `deployment/foo`) — no cluster lookups are performed.