Skip to content

Add agent connection troubleshooting and a log collection guide - #295

Open
hank-metalbear wants to merge 1 commit into
mainfrom
han/daily-agent-connection-docs
Open

Add agent connection troubleshooting and a log collection guide#295
hank-metalbear wants to merge 1 commit into
mainfrom
han/daily-agent-connection-docs

Conversation

@hank-metalbear

Copy link
Copy Markdown
Contributor

Summary

  • Add a troubleshooting section for Unable to connect to agent failures to docs/troubleshooting/common-issues.md, covering the common causes under the exact error strings users see (image pull failures in restricted clusters, startup timeout, node pod capacity, admission/security policies, port-forward path issues), each with the config option or kubectl step that addresses it.
  • Add a new docs/troubleshooting/collecting-logs.md page explaining how to capture layer, internal proxy, and agent logs (the same logs the GitHub bug report template asks for) and what to attach to a report. Linked from the new section, the troubleshooting index, and SUMMARY.

Addresses the remaining asks of COR-176: the troubleshooting section itself already exists, but agent connection failures and log collection guidance were missing from it.

Why this matters for adoption

Agent connection failures are the most common class of error real users hit on first runs, and they disproportionately affect restricted clusters, exactly the environments corporate evaluators run in. When the connection fails today, the CLI's help text only suggests opening a bug or asking on Slack, and no docs page addresses the failure, so many affected users walk away without a diagnosis. A symptom-first triage path (with the literal error strings as headings, so search engines index them) turns silent first-run failures into self-serve fixes, and gives support and solution engineers a link to send instead of a live debugging session.

Test plan

  • Every claim is grounded in the mirrord source: error strings from mirrord/kube/src/error.rs and mirrord/kube/src/api/container/{job,targeted,util}.rs; defaults and option anchors from mirrord/config/src/agent.rs and internal_proxy.rs (startup_timeout default 60, agent pod label app=mirrord, image pull fail-fast on ErrImagePull/ImagePullBackOff, agent.ttl for keeping pods around for logs); log env vars cross-checked against the bug report template and mirrord/layer-lib/src/logging.rs.
  • Verified every external link added here returns 200 via curl, including each config option anchor page; verified relative links point at files that exist in this repo; code fences balanced; no capitalized mirrord.
  • Not exercised: rendering on GitBook itself (this repo has no local build); the pages follow the frontmatter and heading conventions of the existing troubleshooting pages.

🤖 Generated with Claude Code

…176)

Document the common failure modes behind 'Unable to connect to agent'
(image pull, startup timeout, node capacity, security policies, port
forwarding) with the config options that address each, and add a page
on collecting layer, internal proxy, and agent logs for bug reports.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hank-metalbear hank-metalbear self-assigned this Jul 30, 2026
@hank-metalbear
hank-metalbear marked this pull request as ready for review July 30, 2026 16:05
Comment on lines +203 to +205
## mirrord fails with `Unable to connect to agent`

In sessions that don't go through the mirrord Operator, mirrord spawns an agent in the cluster and connects to it by port-forwarding through the Kubernetes API server. `Unable to connect to agent` means one of these steps failed, and the rest of the error message says which one. The most common cases are listed below.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Section heading doesn't match the primary error message users see

In the main mirrord exec flow, agent creation errors (image pull failure, startup timeout, pod capacity) are explicitly mapped to CliError::PortForwardingSetupError in mirrord/cli/src/main.rs (lines 982–989), which carries the message "Failed to make secondary agent connection: …". The error string "Unable to connect to agent" comes from ExecError::AgentConnection in a different code path. A user who sees Failed to make secondary agent connection: Timeout waiting for agent to be ready and searches for that phrase will land here because the H3 sub-heading is correct, but a user searching for the exact H2 phrase won't. Adding a note like "You may also see this as Failed to make secondary agent connection" in the intro paragraph would close the gap without restructuring the page.

Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/troubleshooting/common-issues.md
Line: 203-205

Comment:
**Section heading doesn't match the primary error message users see**

In the main `mirrord exec` flow, agent creation errors (image pull failure, startup timeout, pod capacity) are explicitly mapped to `CliError::PortForwardingSetupError` in `mirrord/cli/src/main.rs` (lines 982–989), which carries the message `"Failed to make secondary agent connection: …"`. The error string `"Unable to connect to agent"` comes from `ExecError::AgentConnection` in a different code path. A user who sees `Failed to make secondary agent connection: Timeout waiting for agent to be ready` and searches for that phrase will land here because the H3 sub-heading is correct, but a user searching for the exact H2 phrase won't. Adding a note like *"You may also see this as `Failed to make secondary agent connection`"* in the intro paragraph would close the gap without restructuring the page.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Codex Fix in Claude Code

Comment on lines +19 to +30
## Internal proxy logs

The internal proxy runs on your machine and relays traffic between the layer and the agent. Its logs go to a file in your temporary directory by default (a path like `/tmp/mirrord-intproxy-<timestamp>-<random>.log`). Raise the level with [`internal_proxy.log_level`](https://metalbear.com/mirrord/docs/config/options#internal_proxy-log_level) (defaults to `mirrord=info,warn`), and pick a fixed location with [`internal_proxy.log_destination`](https://metalbear.com/mirrord/docs/config/options#internal_proxy-log_destination):

```json
{
"internal_proxy": {
"log_level": "mirrord=trace",
"log_destination": "/tmp/intproxy.log"
}
}
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Internal proxy logs default to JSON, not human-readable text

InternalProxyConfig.json_log defaults to true (see mirrord/config/src/internal_proxy.rs line 135). A user who follows this guide and opens the log file will see newline-delimited JSON, not the INFO mirrord::... style lines they might expect. Without this warning, the format can look like noise and the user may think logging isn't working. Consider noting that the default format is JSON and that setting "json_log": false produces human-readable output alongside the log_level example already shown.

Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/troubleshooting/collecting-logs.md
Line: 19-30

Comment:
**Internal proxy logs default to JSON, not human-readable text**

`InternalProxyConfig.json_log` defaults to `true` (see `mirrord/config/src/internal_proxy.rs` line 135). A user who follows this guide and opens the log file will see newline-delimited JSON, not the `INFO mirrord::...` style lines they might expect. Without this warning, the format can look like noise and the user may think logging isn't working. Consider noting that the default format is JSON and that setting `"json_log": false` produces human-readable output alongside the `log_level` example already shown.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Codex Fix in Claude Code

@greptile-apps

greptile-apps Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds two new troubleshooting resources: a symptom-first triage section for Unable to connect to agent failures in common-issues.md, and a new collecting-logs.md page explaining how to capture layer, internal proxy, and agent logs.

  • common-issues.md: Five sub-cases are documented with headings that match the actual error sub-strings emitted by mirrord/kube/src/error.rs and util.rs; config option names (agent.startup_timeout, agent.check_out_of_pods, agent.image, agent.image_pull_secrets, agent.disabled_capabilities, agent.namespace, agent.privileged) were cross-checked against mirrord/config/src/agent.rs and are correct. The startup_timeout default (60 s) matches the source.
  • collecting-logs.md: The MIRRORD_LOG env var, internal_proxy.log_level/log_destination, and agent.log_level config anchors are all verified against the mirrord source. Navigation entries are added correctly to SUMMARY.md and troubleshooting/README.md.

Important Files Changed

Filename Overview
docs/troubleshooting/common-issues.md Adds a new troubleshooting section with five agent-connection sub-cases; error sub-strings and config option names verified against mirrord source, but the H2 heading may not match what users see in the primary exec flow.
docs/troubleshooting/collecting-logs.md New page documenting log collection for layer, internal proxy, and agent; env var names and config anchors check out, but the guide omits that internal proxy logs are JSON by default.
docs/SUMMARY.md Adds Collecting Logs entry in the correct position under the Troubleshooting group; no issues.
docs/troubleshooting/README.md Adds a bullet for the new collecting-logs page to the troubleshooting index; link and description are correct.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[mirrord exec fails] --> B{Error message?}
    B -->|Unable to connect to agent / Failed to make secondary agent connection| C{Sub-error?}
    B -->|Other| Z[See other Common Issues sections]
    C -->|agent container failed to pull image| D[Image pull failure - Use agent.image + agent.image_pull_secrets]
    C -->|Timeout waiting for agent to be ready| E[Startup timeout - Increase agent.startup_timeout]
    C -->|Node X is out of pod capacity| F[Pod capacity - Free node capacity or target different replica]
    C -->|Pod rejected by admission policy| G[Security policy - Use agent.namespace or agent.disabled_capabilities]
    C -->|Connection to agent failed / Port not found| H[Port-forward broken - Check VPN / HTTP proxy / firewall]
    D & E & F & G & H --> I[Collect logs for deeper diagnosis]
    I --> J[Layer: MIRRORD_LOG=mirrord=trace]
    I --> K[Internal proxy: internal_proxy.log_level + log_destination]
    I --> L[Agent: agent.log_level + kubectl logs -l app=mirrord]
Loading

Fix All in Cursor Fix All in Codex Fix All in Claude Code

Prompt To Fix All With AI
### Issue 1
docs/troubleshooting/common-issues.md:203-205
**Section heading doesn't match the primary error message users see**

In the main `mirrord exec` flow, agent creation errors (image pull failure, startup timeout, pod capacity) are explicitly mapped to `CliError::PortForwardingSetupError` in `mirrord/cli/src/main.rs` (lines 982–989), which carries the message `"Failed to make secondary agent connection: …"`. The error string `"Unable to connect to agent"` comes from `ExecError::AgentConnection` in a different code path. A user who sees `Failed to make secondary agent connection: Timeout waiting for agent to be ready` and searches for that phrase will land here because the H3 sub-heading is correct, but a user searching for the exact H2 phrase won't. Adding a note like *"You may also see this as `Failed to make secondary agent connection`"* in the intro paragraph would close the gap without restructuring the page.

### Issue 2
docs/troubleshooting/collecting-logs.md:19-30
**Internal proxy logs default to JSON, not human-readable text**

`InternalProxyConfig.json_log` defaults to `true` (see `mirrord/config/src/internal_proxy.rs` line 135). A user who follows this guide and opens the log file will see newline-delimited JSON, not the `INFO mirrord::...` style lines they might expect. Without this warning, the format can look like noise and the user may think logging isn't working. Consider noting that the default format is JSON and that setting `"json_log": false` produces human-readable output alongside the `log_level` example already shown.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "Add agent connection troubleshooting and..." | Re-trigger Greptile

@hank-metalbear
hank-metalbear requested a review from aviramha July 30, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant