Local runtime: make workload DNS platform-aware and connectivity-verified

[rcyw]

Summary

Make local workload DNS a platform-aware, connectivity-verified runtime capability across supported Axern local environments.

PR #30 fixes the immediate release blocker by discovering non-loopback host resolver IPs, validating them, passing them through the embedded local bundle to axnoded, and exposing a required runtime_dns doctor check. That is the correct near-term boundary: a resolver address that is valid in the outer Node container is not automatically valid inside a nested OCI sandbox.

This issue tracks the remaining long-term product and architecture work. It does not block PR #30 or the current release candidate.

Related: #30

Background

The local Axern stack runs axnoded inside Docker. axnoded then creates nested OCI workloads with their own network namespace.

On a Docker user-defined network, the Node container commonly receives Docker's embedded resolver, such as 127.0.0.11. Copying that address into a nested sandbox is incorrect: loopback is scoped to the sandbox's own network namespace, so it cannot reach the embedded resolver in the outer container.

axnoded intentionally rejects loopback and unspecified resolver addresses. Relaxing that validation would hide the namespace boundary and produce workloads that start successfully but cannot resolve names.

The orchestration/runtime networking layer must therefore materialize DNS configuration that is reachable from the child workload namespace.

Current behavior after PR #30

• axern local discovers non-loopback resolver IPs from supported host resolver files.
• AXERN_LOCAL_DNS_NAMESERVERS provides an explicit comma-separated override.
• Values are normalized, deduplicated, and required to be valid non-loopback, non-unspecified IP addresses.
• The embedded Compose bundle passes the resolved addresses to axnoded through AXNODED_DNS_NAMESERVERS.
• axern local doctor reports runtime_dns as a required check.
• Axern does not hardcode public DNS servers.
• axnoded's strict runtime DNS validation remains unchanged.

This closes the known Docker loopback failure, but static resolver discovery does not prove that a resolver is reachable from the Node container or a nested sandbox.

Problem statement

Local DNS behavior can vary across:

• macOS and Linux;
• Docker Desktop and Docker Engine;
• amd64 and arm64;
• systemd-resolved stub and uplink configurations;
• IPv4-only, IPv6-only, and dual-stack hosts;
• VPN clients and corporate networks;
• split-horizon or per-domain DNS;
• proxies and restricted egress environments.

A syntactically valid resolver IP may still be unreachable from Docker or the nested workload. In addition, a flat nameserver list cannot fully represent every platform's split-DNS routing policy.

The user-facing goal is that axern local up either produces workloads with functional DNS or fails early with a precise, executable remediation. Users should not need to understand Docker's nested network namespaces.

Design principles

1. Preserve namespace correctness

   • Never inject container-local or host-local loopback resolvers into an isolated child namespace unless the runtime deliberately shares that namespace.

2. Keep ownership at the networking boundary

   • Local orchestration discovers platform inputs.
   • The runtime materializes workload DNS configuration.
   • Workloads do not depend directly on Docker, host resolver files, or private control-plane APIs.

3. Do not hardcode public DNS

   • Public resolvers can break private zones, VPNs, compliance requirements, and offline environments.

4. Prefer automatic behavior with an explicit escape hatch

   • Automatic discovery should handle normal installations.
   • AXERN_LOCAL_DNS_NAMESERVERS remains a stable, documented override for managed networks and diagnostics.

5. Verify from the relevant namespace

   • Host-side parsing is a precondition, not proof of runtime reachability.

6. Use the same runtime contract locally and in production

   • Avoid a Compose-only DNS bypass or a local-only private workload API.

7. Fail before the first user Run where practical

   • A required doctor failure is preferable to a workload startup timeout with an indirect OCI error.

Proposed work

Phase 1: connectivity-aware doctor checks

Extend axern local doctor so it distinguishes:

• resolver discovery and syntax validation;
• reachability from the Node container;
• resolution from an isolated test sandbox using the same runtime path as a user Run.

The report should identify the failing layer and provide a direct recommendation. JSON output must expose stable check codes and structured details without leaking sensitive DNS query data.

Candidate checks:

• runtime_dns_config: at least one normalized resolver is available;
• runtime_dns_node: the Node container can query through the configured resolver set;
• runtime_dns_sandbox: a short-lived isolated sandbox can resolve through the materialized runtime configuration.

The exact public check names should be reviewed before stabilization.

Phase 2: supported-platform behavior matrix

Document and test resolver discovery and propagation for:

• macOS + Docker Desktop on amd64/arm64;
• Linux + Docker Engine on amd64/arm64;
• Linux with plain /etc/resolv.conf;
• Linux with systemd-resolved stub plus uplink resolver file;
• IPv4 and IPv6 resolver addresses;
• VPN/split-DNS scenarios where practical in CI or a documented manual qualification environment.

For each platform, define:

• discovery source;
• precedence rules;
• whether search domains and resolver options are preserved;
• expected behavior when the source changes while the local stack is running;
• the remediation shown by local doctor.

Phase 3: evaluate a managed DNS forwarder

Evaluate whether Axern should run or embed a small DNS forwarder reachable from Node and sandbox namespaces.

A forwarder may provide a stable workload-facing resolver while adapting upstream behavior to Docker Desktop, systemd-resolved, VPN, or platform-native DNS. The evaluation must cover:

• split-horizon and per-domain routing support;
• dynamic host/VPN resolver changes;
• UDP and TCP DNS;
• IPv4/IPv6 behavior;
• cache correctness and negative caching;
• DNSSEC expectations;
• lifecycle, health checking, logs, and resource cost;
• attack surface and whether untrusted workloads can abuse it;
• behavior during local up, down, upgrade, and reset;
• whether an existing maintained component is preferable to custom implementation.

Do not adopt a forwarder solely to avoid fixing resolver discovery. It should be introduced only if it materially improves transparent behavior for supported environments.

User experience requirements

• Normal users should not need to set DNS environment variables.
• axern local up should report the selected DNS strategy without exposing unnecessary host details.
• axern local doctor must explain whether failure occurred during discovery, Node connectivity, or sandbox resolution.
• Every required failure must include an actionable remediation, such as setting AXERN_LOCAL_DNS_NAMESERVERS and recreating the Node container.
• Changing DNS configuration must preserve local data.
• local status --output json and local doctor --output json should remain automation-friendly.
• Documentation must cover ordinary networks, VPN/corporate DNS, and safe override examples in English and Simplified Chinese.

Security and privacy requirements

• Reject loopback and unspecified resolver addresses unless a future networking design explicitly makes them reachable.
• Do not silently fall back to a public resolver.
• Do not log full private DNS query names by default.
• Treat resolver addresses and search domains as operational metadata; avoid sending them to telemetry unless explicitly documented and approved.
• Bound diagnostic timeouts and retries so a malicious or unavailable resolver cannot indefinitely block local up or doctor.
• A managed forwarder, if adopted, must bind only to the required local/container networks and must not become an open resolver.

Acceptance criteria

• Resolver discovery behavior and precedence are specified for every supported local platform.
• local doctor differentiates configuration, Node reachability, and sandbox resolution failures.
• A real DNS query is executed through the same DNS configuration used by user workloads.
• Failure completes within a bounded timeout and gives an executable remediation.
• macOS/Linux × amd64/arm64 source-free release smoke validates workload DNS.
• Linux systemd-resolved stub configurations are covered.
• IPv4 and IPv6 resolver inputs are covered or unsupported combinations are rejected clearly.
• VPN/split-DNS behavior is tested or explicitly documented with a supported override path.
• No loopback resolver is accidentally copied across network namespaces.
• No public resolver is hardcoded or used as a silent fallback.
• DNS changes can be applied without deleting PostgreSQL, MinIO objects, identity, or other local data.
• English and Simplified Chinese documentation remain aligned.
• A written decision records whether a managed DNS forwarder is adopted, deferred, or rejected.

Test plan

Unit and component tests

• resolver parsing, normalization, ordering, and deduplication;
• invalid, loopback, unspecified, empty, IPv4, and IPv6 values;
• malformed or partially readable resolver files;
• explicit override precedence;
• doctor check state and JSON schema;
• bounded timeout and cancellation behavior;
• DNS configuration materialized into an OCI bundle.

Integration tests

• Node container can resolve a controlled hostname;
• isolated sandbox can resolve the same hostname;
• resolver failure produces the expected doctor code and recommendation;
• DNS recovery works after local down / local up without data loss;
• workload stdout/stderr and exit status remain correct when DNS succeeds or fails.

Release qualification

On macOS/Linux and amd64/arm64, from a directory without Axern source:

1. install the candidate CLI;
2. run axern local up;
3. run axern local doctor;
4. execute a workload that resolves and connects to a controlled hostname;
5. verify the workload output and exit code;
6. run local down, change the resolver input, and run local up again;
7. verify existing local resources remain available.

Non-goals

• Adding Windows or Podman support as part of this issue.
• Automatically configuring Kubernetes cluster DNS.
• Preserving compatibility with repository-based make quickstart flows.
• Making Axern an authoritative DNS server.
• Supporting arbitrary hostnames in the nameserver list; resolver endpoints remain IP addresses.
• Weakening axnoded validation to accept unreachable loopback addresses.

Delivery guidance

This work can be delivered incrementally:

1. merge PR fix(local): configure workload DNS from host resolvers #30 to unblock the release;
2. add connectivity-aware doctor checks and release smoke coverage;
3. collect evidence from supported platforms and VPN environments;
4. make and record the managed-forwarder decision;
5. implement a forwarder only if the evidence justifies the added lifecycle and security complexity.

[rcyw]

Progress update after the related runtime work:

Completed:

• fix(local): configure workload DNS from host resolvers #30 added platform input discovery for non-loopback resolver addresses, the explicit AXERN_LOCAL_DNS_NAMESERVERS override, propagation into the local axnoded configuration, strict validation, and the initial required runtime_dns doctor check.
• runtime: add EROFS-aware writable rootfs storage contract #41 established /etc/resolv.conf as a runtime-provided OCI sandbox contract, including readonly/immutable lower rootfs handling and runtime-private system-file materialization.

These changes close resolver discovery, propagation, validation, and OCI materialization. They do not complete this issue because syntactically valid resolver addresses are still not proof of connectivity from the Node container or an isolated workload namespace.

Remaining scope:

• split doctor diagnostics into resolver configuration, Node connectivity, and sandbox resolution;
• execute a bounded real DNS query through the same runtime path used by workloads;
• qualify macOS/Linux and amd64/arm64 behavior, including systemd-resolved, IPv4/IPv6, VPN, and split-DNS cases;
• verify DNS changes can be applied without local data loss;
• record the managed DNS forwarder decision and implement one only if qualification evidence justifies it.

Keeping this issue open as the long-term connectivity verification and platform qualification tracker. PRs #30 and #41 should be treated as completed prerequisites, not as closing the remaining acceptance criteria.