Hi team, this is the first of three related observations on the Arc-mode kubelet auth path. They're tightly coupled so I've broken them out into separate issues for clarity and will cross-link them below. Thanks again for v0.1.0 — really enjoying tinkering with it.
What we observed
When the agent is configured with managed-identity kubelet auth (node.kubelet.auth.mode: managedidentity) on a host where the Azure Arc agent (HIMDS) is providing the identity, the exec-credential script the agent writes for kubelet does not appear to propagate the Arc environment variables:
IDENTITY_ENDPOINTIMDS_ENDPOINTIMDS_API_VERSION
In our run, kubelet's exec credential then fell back to the default IMDS at 169.254.169.254 (unreachable on a non-Azure host), the Azure Identity client returned an empty token, kubelet sent an empty Bearer to the AKS API server, and the node never joined. (Happy to share repro logs if useful.)
Repro on the Pi rack
- Arc-onboard a Raspberry Pi 5 (HIMDS active).
- Configure agent with
arc.enabled: true and node.kubelet.auth.mode: managedidentity.
- Start the agent. Kubelet API calls 401 indefinitely;
env from the exec-credential process shows no IDENTITY_ENDPOINT.
Impact for our scenario
This is the path we expected to use for our edge demo, since Arc was already running on the Pis for the agent's own Azure calls. We worked around it by switching kubelet to bootstraptoken, but that path has its own knock-on effects (covered in the two related issues below).
One possible direction
Two options that came to mind, no strong preference:
- Have the exec-credential script
source /etc/default/himds (or whatever Arc places its env in on the host) before invoking the token tool, so the Azure Identity client picks the Arc credential provider.
- Detect Arc presence (e.g.
systemctl is-active himdsd or check /var/opt/azcmagent/) and write the env vars into the exec stanza of the bootstrap kubeconfig directly.
The second is probably nicer because it survives ordering races on cold boot.
Happy to test a prototype on the Pi rack if helpful.
Version tested: v0.1.0 (git 65d8d38).
Related on this auth path (tightly coupled)
Other observations from the same Pi-rack edge demo
Hi team, this is the first of three related observations on the Arc-mode kubelet auth path. They're tightly coupled so I've broken them out into separate issues for clarity and will cross-link them below. Thanks again for v0.1.0 — really enjoying tinkering with it.
What we observed
When the agent is configured with managed-identity kubelet auth (
node.kubelet.auth.mode: managedidentity) on a host where the Azure Arc agent (HIMDS) is providing the identity, the exec-credential script the agent writes for kubelet does not appear to propagate the Arc environment variables:IDENTITY_ENDPOINTIMDS_ENDPOINTIMDS_API_VERSIONIn our run, kubelet's exec credential then fell back to the default IMDS at
169.254.169.254(unreachable on a non-Azure host), the Azure Identity client returned an empty token, kubelet sent an empty Bearer to the AKS API server, and the node never joined. (Happy to share repro logs if useful.)Repro on the Pi rack
arc.enabled: trueandnode.kubelet.auth.mode: managedidentity.envfrom the exec-credential process shows noIDENTITY_ENDPOINT.Impact for our scenario
This is the path we expected to use for our edge demo, since Arc was already running on the Pis for the agent's own Azure calls. We worked around it by switching kubelet to
bootstraptoken, but that path has its own knock-on effects (covered in the two related issues below).One possible direction
Two options that came to mind, no strong preference:
source /etc/default/himds(or whatever Arc places its env in on the host) before invoking the token tool, so the Azure Identity client picks the Arc credential provider.systemctl is-active himdsdor check/var/opt/azcmagent/) and write the env vars into the exec stanza of the bootstrap kubeconfig directly.The second is probably nicer because it survives ordering races on cold boot.
Happy to test a prototype on the Pi rack if helpful.
Version tested: v0.1.0 (git 65d8d38).
Related on this auth path (tightly coupled)
validateExclusiveAuthSettingstreats Arc and bootstrap-token as mutually exclusiveOther observations from the same Pi-rack edge demo
assignRBACRolesvalidateExclusiveAuthSettingstreats Arc and bootstrap-token as mutually exclusivekube1systemd-nspawn launcher passes--network-vethbut actually shares host netns