Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@
* [Troubleshooting](troubleshooting/README.md)
* [Utilities](troubleshooting/utilities.md)
* [Common Issues](troubleshooting/common-issues.md)
* [Collecting Logs](troubleshooting/collecting-logs.md)

## Reference

Expand Down
1 change: 1 addition & 0 deletions docs/troubleshooting/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ description: Common issues and debugging utilities for mirrord

- **[Common Issues](common-issues.md)** - Solutions for frequently encountered problems
- **[Utilities](utilities.md)** - Debugging tools like response header injection and latency diagnosis
- **[Collecting Logs](collecting-logs.md)** - How to capture layer, internal proxy, and agent logs for debugging
57 changes: 57 additions & 0 deletions docs/troubleshooting/collecting-logs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
title: Collecting Logs
date: 2026-07-30T00:00:00.000Z
description: How to capture logs from every part of mirrord (layer, internal proxy, and agent) when debugging or reporting an issue
---

mirrord runs in a few places at once: the layer inside your local process, an internal proxy on your machine, and the agent inside the cluster. Each part logs separately, and the [bug report template](https://github.com/metalbear-co/mirrord/issues/new?assignees=&labels=bug&projects=&template=bug_report.yml) asks for the relevant ones. Here is how to capture each.

## Layer logs

The layer is loaded into your application's process. Enable its logs by setting the `MIRRORD_LOG` environment variable, then run mirrord as usual:

```bash
MIRRORD_LOG=mirrord=trace mirrord exec -- <your command>
```

The layer prints its logs to the application's STDERR, alongside your application's own output. When running through an IDE plugin, set `MIRRORD_LOG` in the run configuration's environment variables; the logs appear in the IDE's run console.

## Internal proxy logs

The internal proxy runs on your machine and relays traffic between the layer and the agent. Its logs go to a file in your temporary directory by default (a path like `/tmp/mirrord-intproxy-<timestamp>-<random>.log`). Raise the level with [`internal_proxy.log_level`](https://metalbear.com/mirrord/docs/config/options#internal_proxy-log_level) (defaults to `mirrord=info,warn`), and pick a fixed location with [`internal_proxy.log_destination`](https://metalbear.com/mirrord/docs/config/options#internal_proxy-log_destination):

```json
{
"internal_proxy": {
"log_level": "mirrord=trace",
"log_destination": "/tmp/intproxy.log"
}
}
```
Comment on lines +19 to +30

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Internal proxy logs default to JSON, not human-readable text

InternalProxyConfig.json_log defaults to true (see mirrord/config/src/internal_proxy.rs line 135). A user who follows this guide and opens the log file will see newline-delimited JSON, not the INFO mirrord::... style lines they might expect. Without this warning, the format can look like noise and the user may think logging isn't working. Consider noting that the default format is JSON and that setting "json_log": false produces human-readable output alongside the log_level example already shown.

Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/troubleshooting/collecting-logs.md
Line: 19-30

Comment:
**Internal proxy logs default to JSON, not human-readable text**

`InternalProxyConfig.json_log` defaults to `true` (see `mirrord/config/src/internal_proxy.rs` line 135). A user who follows this guide and opens the log file will see newline-delimited JSON, not the `INFO mirrord::...` style lines they might expect. Without this warning, the format can look like noise and the user may think logging isn't working. Consider noting that the default format is JSON and that setting `"json_log": false` produces human-readable output alongside the `log_level` example already shown.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Codex Fix in Claude Code


## Agent logs

The agent runs in the cluster, as a `mirrord-agent-*` pod in the target's namespace (or as an ephemeral container inside the target pod). Raise its log level with [`agent.log_level`](https://metalbear.com/mirrord/docs/config/options#agent-log_level) and read the logs with kubectl:

```bash
kubectl logs -n <namespace> -l app=mirrord
```

Agent pods are cleaned up shortly after the session ends. If the pod disappears before you can read the logs, keep it around longer with [`agent.ttl`](https://metalbear.com/mirrord/docs/config/options#agent-ttl), which controls how many seconds the pod persists after the agent exits:

```json
{
"agent": {
"log_level": "mirrord=trace",
"ttl": 300
}
}
```

## Operator logs

If you use mirrord for Teams, the Operator produces its own logs in the cluster. See [Monitoring](../managing-mirrord/monitoring.md) for how to configure and collect them.

## What to attach to a report

For most issues, layer logs and internal proxy logs from a failed run are enough. For agent startup and connection problems, add the agent pod's logs and the output of `kubectl describe` on the agent pod. Strip anything sensitive before sharing; log lines can include hostnames, paths, and header names from your environment.
60 changes: 60 additions & 0 deletions docs/troubleshooting/common-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,66 @@ By excluding `NX_NEXT_DIR` and `NODE_ENV`, you stop the remote environment from

---

## mirrord fails with `Unable to connect to agent`

In sessions that don't go through the mirrord Operator, mirrord spawns an agent in the cluster and connects to it by port-forwarding through the Kubernetes API server. `Unable to connect to agent` means one of these steps failed, and the rest of the error message says which one. The most common cases are listed below.
Comment on lines +203 to +205

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Section heading doesn't match the primary error message users see

In the main mirrord exec flow, agent creation errors (image pull failure, startup timeout, pod capacity) are explicitly mapped to CliError::PortForwardingSetupError in mirrord/cli/src/main.rs (lines 982–989), which carries the message "Failed to make secondary agent connection: …". The error string "Unable to connect to agent" comes from ExecError::AgentConnection in a different code path. A user who sees Failed to make secondary agent connection: Timeout waiting for agent to be ready and searches for that phrase will land here because the H3 sub-heading is correct, but a user searching for the exact H2 phrase won't. Adding a note like "You may also see this as Failed to make secondary agent connection" in the intro paragraph would close the gap without restructuring the page.

Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/troubleshooting/common-issues.md
Line: 203-205

Comment:
**Section heading doesn't match the primary error message users see**

In the main `mirrord exec` flow, agent creation errors (image pull failure, startup timeout, pod capacity) are explicitly mapped to `CliError::PortForwardingSetupError` in `mirrord/cli/src/main.rs` (lines 982–989), which carries the message `"Failed to make secondary agent connection: …"`. The error string `"Unable to connect to agent"` comes from `ExecError::AgentConnection` in a different code path. A user who sees `Failed to make secondary agent connection: Timeout waiting for agent to be ready` and searches for that phrase will land here because the H3 sub-heading is correct, but a user searching for the exact H2 phrase won't. Adding a note like *"You may also see this as `Failed to make secondary agent connection`"* in the intro paragraph would close the gap without restructuring the page.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Codex Fix in Claude Code


For anything the error message doesn't explain, the agent pod is the best witness. It is created in the target's namespace (or in [`agent.namespace`](https://metalbear.com/mirrord/docs/config/options#agent-namespace) if set) with the `app=mirrord` label:

```bash
kubectl get pods -n <namespace> -l app=mirrord
kubectl describe pod -n <namespace> <mirrord-agent-pod-name>
```

The events at the bottom of `kubectl describe` explain most scheduling and image problems.

### `agent container failed to pull image (ErrImagePull or ImagePullBackOff)`

The cluster nodes cannot pull the agent image. By default mirrord uses the public image `ghcr.io/metalbear-co/mirrord`, so this typically happens in clusters that cannot access public registries (air-gapped environments, restricted egress). Copy the agent image to a registry your cluster can reach, then point mirrord at it with [`agent.image`](https://metalbear.com/mirrord/docs/config/options#agent-image), adding [`agent.image_pull_secrets`](https://metalbear.com/mirrord/docs/config/options#agent-image_pull_secrets) if the registry requires authentication:

```json
{
"agent": {
"image": "internal.registry.example.com/mirrord:latest",
"image_pull_secrets": [{ "name": "my-registry-secret" }]
}
}
```

### `Timeout waiting for agent to be ready`

The agent did not finish starting within [`agent.startup_timeout`](https://metalbear.com/mirrord/docs/config/options#agent-startup_timeout) (60 seconds by default). This is usually a pod that is still scheduling, or a node pulling the agent image for the first time. Check the pod events as shown above. If things are just slow, raise the timeout:

```json
{
"agent": {
"startup_timeout": 300
}
}
```

### `Node <name> is out of pod capacity`

When targeting a workload, the agent pod runs on the same node as the target pod, because it works directly with the target's container on that node. If that node already runs its maximum number of pods, there is no room for the agent. Free some capacity on that node, or target a replica that runs on a less crowded node. mirrord performs this capacity check before creating the agent; [`agent.check_out_of_pods`](https://metalbear.com/mirrord/docs/config/options#agent-check_out_of_pods) can disable the check itself, but if the node truly has no capacity, the agent will still fail to schedule.

### The agent pod is blocked by cluster security policies

The agent requests elevated Linux capabilities in order to work with the target's network and processes (see [`agent.disabled_capabilities`](https://metalbear.com/mirrord/docs/config/options#agent-disabled_capabilities) for the exact list). Admission policies that restrict capabilities can reject the agent pod or keep it from scheduling:

- OpenShift's default security context constraints (see the [limitations FAQ](../faq/limitations.md))
- Namespaces enforcing the `restricted` [Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/)
- Policy engines such as OPA Gatekeeper or Kyverno

`kubectl describe` on the pod, or the policy engine's own events, shows the rejection reason. Depending on your setup you can run the agent in a dedicated, less restricted namespace with [`agent.namespace`](https://metalbear.com/mirrord/docs/config/options#agent-namespace) (not available for targetless runs or ephemeral agents), or disable capabilities you don't need with [`agent.disabled_capabilities`](https://metalbear.com/mirrord/docs/config/options#agent-disabled_capabilities) (the features that rely on them stop working). On hardened node operating systems such as Bottlerocket, the agent may additionally need [`agent.privileged`](https://metalbear.com/mirrord/docs/config/options#agent-privileged). If your organization prefers keeping strict policies as they are, [mirrord for Teams](../managing-mirrord/operator.md) moves agent creation to the mirrord Operator, so cluster admins grant the needed permissions once instead of to every developer.

### Connection drops or port-forward failures

mirrord reaches the agent through the same path as `kubectl port-forward`. If you see errors like `Connection to agent failed` or `Port not found in port forward`, first verify that `kubectl port-forward` works against any pod in the cluster. If it doesn't, fix that first: VPNs, corporate HTTP proxies, and private cluster firewalls commonly break the port-forward path while leaving other kubectl commands working.

If none of these match your case, collect the [mirrord logs](collecting-logs.md) from the failed run before reaching out; they make the difference between a guess and a fix.

---

## Didn't find your issue here?

- Search or ask on our [Slack community](https://metalbear.com/slack)
Expand Down