Add agent connection troubleshooting and a log collection guide - #295
Add agent connection troubleshooting and a log collection guide#295hank-metalbear wants to merge 1 commit into
Conversation
…176) Document the common failure modes behind 'Unable to connect to agent' (image pull, startup timeout, node capacity, security policies, port forwarding) with the config options that address each, and add a page on collecting layer, internal proxy, and agent logs for bug reports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ## mirrord fails with `Unable to connect to agent` | ||
|
|
||
| In sessions that don't go through the mirrord Operator, mirrord spawns an agent in the cluster and connects to it by port-forwarding through the Kubernetes API server. `Unable to connect to agent` means one of these steps failed, and the rest of the error message says which one. The most common cases are listed below. |
There was a problem hiding this comment.
Section heading doesn't match the primary error message users see
In the main mirrord exec flow, agent creation errors (image pull failure, startup timeout, pod capacity) are explicitly mapped to CliError::PortForwardingSetupError in mirrord/cli/src/main.rs (lines 982–989), which carries the message "Failed to make secondary agent connection: …". The error string "Unable to connect to agent" comes from ExecError::AgentConnection in a different code path. A user who sees Failed to make secondary agent connection: Timeout waiting for agent to be ready and searches for that phrase will land here because the H3 sub-heading is correct, but a user searching for the exact H2 phrase won't. Adding a note like "You may also see this as Failed to make secondary agent connection" in the intro paragraph would close the gap without restructuring the page.
Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/troubleshooting/common-issues.md
Line: 203-205
Comment:
**Section heading doesn't match the primary error message users see**
In the main `mirrord exec` flow, agent creation errors (image pull failure, startup timeout, pod capacity) are explicitly mapped to `CliError::PortForwardingSetupError` in `mirrord/cli/src/main.rs` (lines 982–989), which carries the message `"Failed to make secondary agent connection: …"`. The error string `"Unable to connect to agent"` comes from `ExecError::AgentConnection` in a different code path. A user who sees `Failed to make secondary agent connection: Timeout waiting for agent to be ready` and searches for that phrase will land here because the H3 sub-heading is correct, but a user searching for the exact H2 phrase won't. Adding a note like *"You may also see this as `Failed to make secondary agent connection`"* in the intro paragraph would close the gap without restructuring the page.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| ## Internal proxy logs | ||
|
|
||
| The internal proxy runs on your machine and relays traffic between the layer and the agent. Its logs go to a file in your temporary directory by default (a path like `/tmp/mirrord-intproxy-<timestamp>-<random>.log`). Raise the level with [`internal_proxy.log_level`](https://metalbear.com/mirrord/docs/config/options#internal_proxy-log_level) (defaults to `mirrord=info,warn`), and pick a fixed location with [`internal_proxy.log_destination`](https://metalbear.com/mirrord/docs/config/options#internal_proxy-log_destination): | ||
|
|
||
| ```json | ||
| { | ||
| "internal_proxy": { | ||
| "log_level": "mirrord=trace", | ||
| "log_destination": "/tmp/intproxy.log" | ||
| } | ||
| } | ||
| ``` |
There was a problem hiding this comment.
Internal proxy logs default to JSON, not human-readable text
InternalProxyConfig.json_log defaults to true (see mirrord/config/src/internal_proxy.rs line 135). A user who follows this guide and opens the log file will see newline-delimited JSON, not the INFO mirrord::... style lines they might expect. Without this warning, the format can look like noise and the user may think logging isn't working. Consider noting that the default format is JSON and that setting "json_log": false produces human-readable output alongside the log_level example already shown.
Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/troubleshooting/collecting-logs.md
Line: 19-30
Comment:
**Internal proxy logs default to JSON, not human-readable text**
`InternalProxyConfig.json_log` defaults to `true` (see `mirrord/config/src/internal_proxy.rs` line 135). A user who follows this guide and opens the log file will see newline-delimited JSON, not the `INFO mirrord::...` style lines they might expect. Without this warning, the format can look like noise and the user may think logging isn't working. Consider noting that the default format is JSON and that setting `"json_log": false` produces human-readable output alongside the `log_level` example already shown.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
|
| Filename | Overview |
|---|---|
| docs/troubleshooting/common-issues.md | Adds a new troubleshooting section with five agent-connection sub-cases; error sub-strings and config option names verified against mirrord source, but the H2 heading may not match what users see in the primary exec flow. |
| docs/troubleshooting/collecting-logs.md | New page documenting log collection for layer, internal proxy, and agent; env var names and config anchors check out, but the guide omits that internal proxy logs are JSON by default. |
| docs/SUMMARY.md | Adds Collecting Logs entry in the correct position under the Troubleshooting group; no issues. |
| docs/troubleshooting/README.md | Adds a bullet for the new collecting-logs page to the troubleshooting index; link and description are correct. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[mirrord exec fails] --> B{Error message?}
B -->|Unable to connect to agent / Failed to make secondary agent connection| C{Sub-error?}
B -->|Other| Z[See other Common Issues sections]
C -->|agent container failed to pull image| D[Image pull failure - Use agent.image + agent.image_pull_secrets]
C -->|Timeout waiting for agent to be ready| E[Startup timeout - Increase agent.startup_timeout]
C -->|Node X is out of pod capacity| F[Pod capacity - Free node capacity or target different replica]
C -->|Pod rejected by admission policy| G[Security policy - Use agent.namespace or agent.disabled_capabilities]
C -->|Connection to agent failed / Port not found| H[Port-forward broken - Check VPN / HTTP proxy / firewall]
D & E & F & G & H --> I[Collect logs for deeper diagnosis]
I --> J[Layer: MIRRORD_LOG=mirrord=trace]
I --> K[Internal proxy: internal_proxy.log_level + log_destination]
I --> L[Agent: agent.log_level + kubectl logs -l app=mirrord]
Prompt To Fix All With AI
### Issue 1
docs/troubleshooting/common-issues.md:203-205
**Section heading doesn't match the primary error message users see**
In the main `mirrord exec` flow, agent creation errors (image pull failure, startup timeout, pod capacity) are explicitly mapped to `CliError::PortForwardingSetupError` in `mirrord/cli/src/main.rs` (lines 982–989), which carries the message `"Failed to make secondary agent connection: …"`. The error string `"Unable to connect to agent"` comes from `ExecError::AgentConnection` in a different code path. A user who sees `Failed to make secondary agent connection: Timeout waiting for agent to be ready` and searches for that phrase will land here because the H3 sub-heading is correct, but a user searching for the exact H2 phrase won't. Adding a note like *"You may also see this as `Failed to make secondary agent connection`"* in the intro paragraph would close the gap without restructuring the page.
### Issue 2
docs/troubleshooting/collecting-logs.md:19-30
**Internal proxy logs default to JSON, not human-readable text**
`InternalProxyConfig.json_log` defaults to `true` (see `mirrord/config/src/internal_proxy.rs` line 135). A user who follows this guide and opens the log file will see newline-delimited JSON, not the `INFO mirrord::...` style lines they might expect. Without this warning, the format can look like noise and the user may think logging isn't working. Consider noting that the default format is JSON and that setting `"json_log": false` produces human-readable output alongside the `log_level` example already shown.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "Add agent connection troubleshooting and..." | Re-trigger Greptile
Summary
Unable to connect to agentfailures todocs/troubleshooting/common-issues.md, covering the common causes under the exact error strings users see (image pull failures in restricted clusters, startup timeout, node pod capacity, admission/security policies, port-forward path issues), each with the config option or kubectl step that addresses it.docs/troubleshooting/collecting-logs.mdpage explaining how to capture layer, internal proxy, and agent logs (the same logs the GitHub bug report template asks for) and what to attach to a report. Linked from the new section, the troubleshooting index, and SUMMARY.Addresses the remaining asks of COR-176: the troubleshooting section itself already exists, but agent connection failures and log collection guidance were missing from it.
Why this matters for adoption
Agent connection failures are the most common class of error real users hit on first runs, and they disproportionately affect restricted clusters, exactly the environments corporate evaluators run in. When the connection fails today, the CLI's help text only suggests opening a bug or asking on Slack, and no docs page addresses the failure, so many affected users walk away without a diagnosis. A symptom-first triage path (with the literal error strings as headings, so search engines index them) turns silent first-run failures into self-serve fixes, and gives support and solution engineers a link to send instead of a live debugging session.
Test plan
mirrord/kube/src/error.rsandmirrord/kube/src/api/container/{job,targeted,util}.rs; defaults and option anchors frommirrord/config/src/agent.rsandinternal_proxy.rs(startup_timeout default 60, agent pod labelapp=mirrord, image pull fail-fast on ErrImagePull/ImagePullBackOff, agent.ttl for keeping pods around for logs); log env vars cross-checked against the bug report template andmirrord/layer-lib/src/logging.rs.🤖 Generated with Claude Code