Hi team, third of three related observations on the Arc-mode kubelet auth path. This one is more enhancement than bug. Cross-links at the bottom.
What we observed
In Arc, MI and SP modes the agent already holds Azure credentials that can call the AKS RP. But in none of those modes does it appear to mint a kubelet bootstrap-token on the user's behalf. The operator has to either:
- Run
kubectl create token (or apply a bootstrap-token Secret YAML) themselves and paste it into the agent config, or
- Rely on the MI exec-credential path (separate issue).
Impact for our scenario
The shape of the customer story we're building toward is "operator stands in front of an edge device, runs one command, walks away." Asking them to also touch kubectl on the cloud side and paste a token back is an awkward handoff, since the agent already has all the ingredients to do it for them.
One possible direction
When bootstrapToken is unset and the agent has Azure creds that can call the cluster, the agent could mint one itself:
- Call
listClusterAdminCredentials to get an admin kubeconfig (this Azure call already exists in v0.1.0).
- Use it to create a bootstrap-token secret in
kube-system with a short TTL (e.g. 1h, just enough for kubelet to come up and request its long-term cert).
- Pass the token straight to kubelet's bootstrap kubeconfig.
This would also let folks Arc-onboard a device with no kubectl access at all, which is the common edge case. Worth documenting the required Azure RBAC scope clearly (Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action).
Happy to test a prototype on the Pi rack.
Version tested: v0.1.0 (git 65d8d38).
Related on this auth path (tightly coupled)
Other observations from the same Pi-rack edge demo
Hi team, third of three related observations on the Arc-mode kubelet auth path. This one is more enhancement than bug. Cross-links at the bottom.
What we observed
In Arc, MI and SP modes the agent already holds Azure credentials that can call the AKS RP. But in none of those modes does it appear to mint a kubelet bootstrap-token on the user's behalf. The operator has to either:
kubectl create token(or apply a bootstrap-token Secret YAML) themselves and paste it into the agent config, orImpact for our scenario
The shape of the customer story we're building toward is "operator stands in front of an edge device, runs one command, walks away." Asking them to also touch
kubectlon the cloud side and paste a token back is an awkward handoff, since the agent already has all the ingredients to do it for them.One possible direction
When
bootstrapTokenis unset and the agent has Azure creds that can call the cluster, the agent could mint one itself:listClusterAdminCredentialsto get an admin kubeconfig (this Azure call already exists in v0.1.0).kube-systemwith a short TTL (e.g. 1h, just enough for kubelet to come up and request its long-term cert).This would also let folks Arc-onboard a device with no
kubectlaccess at all, which is the common edge case. Worth documenting the required Azure RBAC scope clearly (Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action).Happy to test a prototype on the Pi rack.
Version tested: v0.1.0 (git 65d8d38).
Related on this auth path (tightly coupled)
validateExclusiveAuthSettingstreats Arc and bootstrap-token as mutually exclusiveOther observations from the same Pi-rack edge demo
assignRBACRolesvalidateExclusiveAuthSettingstreats Arc and bootstrap-token as mutually exclusivekube1systemd-nspawn launcher passes--network-vethbut actually shares host netns