feat: CommunityToolkit.Aspire.Hosting.K3s — k3s Kubernetes cluster hosting integration by edmondshtogu · Pull Request #1322 · CommunityToolkit/Aspire

edmondshtogu · 2026-05-14T10:45:04Z

Closes #1321

Overview of changes

Adds CommunityToolkit.Aspire.Hosting.K3s, a hosting integration that runs a lightweight Kubernetes cluster as an Aspire resource tree. Developers can declare a local Kubernetes cluster in Program.cs — with Helm charts, manifests, and exposed service endpoints — the same way they add Redis or PostgreSQL. No external tooling beyond a compatible container runtime (Docker or Podman) is required.

What's included

New package: src/CommunityToolkit.Aspire.Hosting.K3s/

File	Responsibility
K3sClusterResource.cs	ContainerResource — k3s server; holds kubeconfig directory path and image settings
K3sClusterOptions.cs	Configuration (pod/service CIDR, disabled components, k3s image tag, helm/kubectl image overrides)
K3sBuilderExtensions.cs	AddK3sCluster, WithDataVolume, WithLifetime, WithReference(cluster), WithK3sVersion, …
K3sReadinessHealthCheck.cs	File-based health check — polls cluster/kubeconfig.yaml, writes local/ + container/ variants, probes nodes via KubernetesClient
HelmReleaseResource.cs	ContainerResource — runs alpine/helm; child of cluster; exits 0 on success
K3sBuilderExtensions.Helm.cs	AddHelmRelease, WithHelmValue, WithHelmValuesFile
K8sManifestResource.cs	ContainerResource — runs alpine/k8s; child of cluster; exits 0 on success
K3sBuilderExtensions.Manifest.cs	AddK8sManifest with auto-detected Kustomize support
K3sServiceEndpointResource.cs	Resource — in-process port-forward; M1 passive health via IsReady flag
K3sBuilderExtensions.ServiceEndpoint.cs	AddServiceEndpoint, WithReference(endpoint)
K3sInProcessPortForwarder.cs	KubernetesClient WebSocket TCP forwarder; binds 0.0.0.0:{port}
K3sAgentResource.cs	Worker node support (K3sClusterOptions.AgentCount)
HelmContainerImageTags.cs / KubectlContainerImageTags.cs	Pinned image defaults; overridable via K3sClusterOptions

New Tests & Examples:

Unit Tests: tests/CommunityToolkit.Aspire.Hosting.K3s.Tests/ — 87 unit tests covering resource registration, script generation, kubeconfig variants, Kustomize detection, values file injection, and public API null guards.
Examples: examples/k3s/CommunityToolkit.Aspire.Hosting.K3s.AppHost/
TypeScript playground: playground/polyglot/TypeScript/CommunityToolkit.Aspire.Hosting.K3s/ValidationAppHost/

Key design decisions

Health check via bind-mount, not docker exec. k3s writes its kubeconfig to K3S_KUBECONFIG_OUTPUT=/tmp/k3s-kubeconfig/kubeconfig.yaml, bind-mounted to AppHostDirectory/.k3s/{name}/cluster/ on the host. The health check polls File.Exists, rewrites server URLs into local/ and container/ variants, then confirms node readiness via IKubernetes.CoreV1.ListNodeAsync. No shell access, no docker exec, works with any container runtime.
Helm and kubectl run as containers. HelmReleaseResource and K8sManifestResource extend ContainerResource and are shown as children of the cluster in the Aspire dashboard. The install/apply script is injected via WithContainerFiles. They cannot use WaitFor(cluster) (Aspire forbids a child waiting for its parent), so their scripts poll for /root/.kube/kubeconfig.yaml — which only appears after the cluster health check passes — before proceeding. Consumers use WaitForCompletion(helmRelease) since these are run-to-completion containers.
Kubeconfig delivered via bind-mount to all containers. WithReference(cluster) on containers, the helm installer, and the kubectl applier all bind-mount AppHostDirectory/.k3s/{name}/container/ (server: https://{name}:6443) at a known in-container path and set KUBECONFIG. Bind-mount is used uniformly so the kubeconfig updates automatically if the cluster is recreated without restarting dependent containers. No KUBECONFIG_DATA base64 encoding — all standard Kubernetes tooling (kubectl, helm, KubernetesClient SDK) works without custom bootstrap code.
Kustomize auto-detected. AddK8sManifest checks for kustomization.yaml at configuration time: if present, it bind-mounts the directory (preserving relative base references) and uses kubectl apply -k; otherwise it uses an async WithContainerFiles callback to copy only the YAML files at container-start time and applies with kubectl apply -f --server-side. The callback approach avoids Aspire's build-time path validation on the string overload. The script auto-detects the mode at runtime.
Service exposure without NodePort. K3sServiceEndpointResource starts an in-process KubernetesClient WebSocket port-forward bound to 0.0.0.0:{hostPort}. Host resources receive services__{name}__url=http(s)://localhost:{port}; container resources receive services__{name}__url=http(s)://host.docker.internal:{port} with --add-host=host.docker.internal:host-gateway injected automatically via ContainerRuntimeArgsCallbackAnnotation (DCP does not inject this on Linux). The forwarder resolves targetPort from the service spec (not the service port) before opening the pod WebSocket, and only signals ready after a running pod is confirmed — not when the TCP listener starts.
Image overrides via K3sClusterOptions. The helm and kubectl container images are configurable via HelmImage/HelmTag/HelmRegistry and KubectlImage/KubectlTag/KubectlRegistry on K3sClusterOptions. Defaults: docker.io/alpine/helm:3.17.3 and docker.io/alpine/kubectl:1.36.0.
Robustness fixes from review. WithK3sVersion propagates the image tag to all agent nodes (prevents server/agent version skew). AddServiceEndpoint validates the port is in the range 1–65535. Helm values files are indexed as {i}-{filename} so declaration order is preserved and basename collisions are safe. All helm --set values and --values paths are POSIX single-quote escaped via ShellEscape().

Usage

var cluster = builder.AddK3sCluster("k8s")
    .WithDataVolume()
    .WithK3sVersion("v1.36.0-k3s1");

var widgetCrd = cluster.AddK8sManifest("widget-crd", "./k8s/crds/");

var argocd = cluster.AddHelmRelease("argocd", "argo-cd",
    repo: "https://argoproj.github.io/argo-helm",
    version: "7.8.0",
    @namespace: "argocd")
    .WithHelmValuesFile("./deploy/argocd-values.yaml");

var ui = cluster.AddServiceEndpoint("argocd-ui", "argocd-server", 443, "argocd")
    .WaitForCompletion(argocd);

builder.AddProject<Projects.WidgetOperator>("operator")
    .WaitForCompletion(widgetCrd)
    .WithReference(cluster);

builder.AddProject<Projects.Api>("api")
    .WaitFor(ui)
    .WithReference(ui);

Example

PR Checklist

Other information

Security Assumptions
This change assumes local development only. k3s runs in privileged mode (required by k3s/containerd). The bind-mounted kubeconfig directory is readable only by the AppHost user. KubernetesClientConfiguration uses the embedded CA cert from the kubeconfig; no DangerousAcceptAnyServerCertificateValidator is used.

Remaining Follow-up Work

Publish-time diagnostic when a K3sClusterResource has no production counterpart configured.

github-actions · 2026-05-14T10:45:13Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 1322

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 1322"

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

edmondshtogu · 2026-05-14T11:16:38Z

@dotnet-policy-service agree

Copilot

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 13 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 15 comments.

Comments suppressed due to low confidence (4)

src/CommunityToolkit.Aspire.Hosting.K3s/K3sBuilderExtensions.Manifest.cs:95

This async callback has no await, which will produce CS1998 and be treated as an error in this repo. Use a completed Task return (or a synchronous overload) so the project builds cleanly.

            resourceBuilder.WithContainerFiles("/k8s-manifests", async (ctx, ct) =>
            {
                if (Directory.Exists(absolutePath))

src/CommunityToolkit.Aspire.Hosting.K3s/K3sBuilderExtensions.Helm.cs:91

This async expression-bodied callback has no await, so CS1998 will be emitted and treated as an error. Return a completed task (or use a synchronous overload) instead of marking the lambda async.

            .WithContainerFiles("/helm-values", async (ctx, ct) =>
                release.ValuesFiles
                    .Select((hostPath, i) => (ContainerFileSystemItem)new ContainerFile

src/CommunityToolkit.Aspire.Hosting.K3s/K3sInProcessPortForwarder.cs:152

The same empty-selector case here can select the first ready pod in the namespace and forward traffic to an unrelated workload. Services without selectors are valid in Kubernetes, so the forwarder should not treat an empty selector as "all pods".

            var selector = string.Join(",",
                (svc.Spec.Selector ?? new Dictionary<string, string>()).Select(kv => $"{kv.Key}={kv.Value}"));

            var pods = await k8sClient.CoreV1
                .ListNamespacedPodAsync(@namespace, labelSelector: selector, cancellationToken: ct)

tests/CommunityToolkit.Aspire.Hosting.K3s.Tests/K3sClusterResourceTests.cs:253

This test name says it verifies the pod subnet argument, but the assertion only checks that a cluster resource exists. It would still pass if WithPodSubnet stopped adding --cluster-cidr, so it should assert the command-line args.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 7 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…isolation - Resolve .gitignore merge conflict markers (keep both sides) - Register K3sReadinessHealthCheck as a singleton to prevent per-tick client leaks - Write kubeconfig variants atomically via temp-file-then-rename - Fix stale credential recovery to not delete the raw kubeconfig k3s will not rewrite - Fix AllocatePort to probe on IPAddress.Any and re-allocate a fresh port on retry - Make WithDataVolume and WithReference(cluster) idempotent against duplicate calls - Apply HelmEscape to --set keys as well as values - Mark kubeconfig bind-mounts as read-only - Throw InvalidOperationException on no-selector services instead of silently marking ready - Add whitespace validation on serviceName/namespace in AddServiceEndpoint - Remove WithHttpsDeveloperCertificate from k3s container (no-op for k3s) - Extract K3sFileHelpers constant for 0755 script mode and /tmp/k3s-kubeconfig.yaml path - Switch kubeconfig container mount from directory to file-level to isolate kubectl cache

Health check (K3sReadinessHealthCheck): - Drop _cachedClient — registration was already fixed to singleton so the cache was live, but a fresh Kubernetes client per check is simpler, removes all stale-client invalidation complexity, and the TLS overhead every 5 s is negligible for local dev - Extract EnsureKubeconfigVariantsAsync; stale detection now only deletes the derived local/ and container/ files — rawPath is never touched Port forwarder (K3sInProcessPortForwarder): - Implement IAsyncDisposable; own a CancellationTokenSource and TcpListener as fields so DisposeAsync can cancel and release the socket independently of the outer application-lifetime token - RunAsync links _cts.Token with the outer ct so either side can stop the loop - Per-connection Task.Run now uses the linked token instead of CancellationToken.None — connections can be cancelled on shutdown - On listener failure a fresh port is re-allocated for the retry Service endpoint wiring: - K3sServiceEndpointResource retains the Forwarder reference - RunEndpointAsync wraps forwarder in await using so DisposeAsync runs on any exit path - Subscribe to ResourceStoppedEvent on the cluster to dispose the forwarder immediately when the cluster container stops, releasing the host port without waiting for the application-lifetime token

IEvangelist

This LGTM, I'm not going to merge it though as this really belongs to @aaronpowell — I'll leave that to him. Thank you for the PR though, @edmondshtogu.

edmondshtogu · 2026-05-20T17:54:22Z

Thanks @IEvangelist, also test I added are passing, the failing ones are not related with my changes.

davidfowl · 2026-06-02T14:32:10Z

Did we verify that the TypeScript path works end to end?

edmondshtogu · 2026-06-02T17:22:35Z

Did we verify that the TypeScript path works end to end?

@davidfowl yes, I introduced additional changes since the callback (Action) was unusable in TypeScript polyglot apphosts — the generated Promise-based wrapper never resolved, making agentCount and the Helm/kubectl image overrides silently unreachable. Here is the TypeScript polyglot in action:

edmondshtogu · 2026-06-05T08:17:32Z

Can someone approve the tests so I can see if there is anything else to be improved?

edmondshtogu · 2026-06-08T12:04:22Z

Can someone approve the workflows again so I can see if there is anything else to be improved?

Important

The timheuer/setup-aspire action installs the latest available CLI build in staging channel, which may be a newer patch than the version pinned in Directory.Build.props. The previous check required an exact match (or the exact version with a +sha suffix), so it would fail whenever the action installed 13.4.3+abc while the repo expected 13.4.0 — even though a newer patch is always compatible. I introduced a fix to strips build metadata before comparing and allow any installed version ≥ the expected version. The .Net main Workflow now is passing!

EndpointReference.IsAllocated caches its result on first call via a nullable bool field (??= pattern). For persistent containers in polyglot AppHosts, the health check can tick before DCP fires the endpoint allocation event, permanently caching false and making the port unresolvable for the lifetime of the process — leaving local/kubeconfig.yaml stale with port 6443. Replace the IsAllocated guard + GetValueAsync path with a direct read of EndpointAnnotation.AllocatedEndpoint.Port. This property is non-cached (checks IsValueSet on every call) and non-blocking (returns null if DCP has not yet allocated), so it picks up DCP's allocation on any subsequent health check tick regardless of when the first tick occurred. Port resolution order: 1. annotation.AllocatedEndpoint.Port — set by DCP on container start 2. annotation.Port — static apiServerPort from AddK3sCluster Remove the EndpointReference constructor parameter and the port hint file mechanism, both of which are unnecessary with this approach.

edmondshtogu · 2026-06-10T15:19:31Z

@aaronpowell could you please take a look at the PR and share if there is something you would like to change, or if you are happy to approve the integration? I'd love to start using this in my projects. Currently, because microsoft/aspire#16878 is still open, I can't add third-party integrations via the Aspire CLI, so having this package available is the only way around.

edmondshtogu added 3 commits May 13, 2026 18:50

feat: CommunityToolkit.Aspire.Hosting.K3s

983b1cd

feat: proper support for service endpoints, helm and manifests resources

4dda24b

feat: support files for helm/manifest

a2ce723

Copilot AI review requested due to automatic review settings May 14, 2026 10:45

Copilot started reviewing on behalf of edmondshtogu May 14, 2026 10:45 View session

Copilot AI reviewed May 14, 2026

edmondshtogu mentioned this pull request May 14, 2026

feat: K3s Kubernetes cluster hosting integration #1321

Open

edmondshtogu added 4 commits May 14, 2026 14:07

Merge branch 'main' into main

2729a69

Merge branch 'main' into main

bc271ec

Merge branch 'main' into main

0e5c9de

compliting left work

d343ee3

edmondshtogu requested a review from Copilot May 18, 2026 12:52

Copilot started reviewing on behalf of edmondshtogu May 18, 2026 12:52 View session

Copilot AI reviewed May 18, 2026

View reviewed changes

edmondshtogu and others added 3 commits May 18, 2026 14:59

upgrade example aspire version

0a4d13a

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix: copilot reviews

f9a50d1

chore: exluding integration tests for win runners

da4bf9d

edmondshtogu requested a review from Copilot May 18, 2026 14:59

Copilot started reviewing on behalf of edmondshtogu May 18, 2026 15:00 View session

Copilot AI reviewed May 18, 2026

View reviewed changes

edmondshtogu and others added 2 commits May 18, 2026 17:10

chore: update example comment

ff66be4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix: copilot reviews

37c11e3

edmondshtogu requested a review from Copilot May 18, 2026 17:57

Copilot started reviewing on behalf of edmondshtogu May 18, 2026 17:58 View session

Copilot AI reviewed May 18, 2026

View reviewed changes

edmondshtogu and others added 3 commits May 18, 2026 20:15

fix: WaitForServiceReadyAsync service health check

99297cd

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

chore: addressing reviews

7d34ddc

fix: copilot reviews

7473170

edmondshtogu added 2 commits May 20, 2026 17:47

edmondshtogu requested a review from IEvangelist May 20, 2026 16:48

fix: manifest apply exit code when no kustomize present

06a7b0f

IEvangelist reviewed May 20, 2026

View reviewed changes

Comment thread examples/k3s/CommunityToolkit.Aspire.Hosting.K3s.AppHost/k8s/monitoring/kustomization.yaml

IEvangelist reviewed May 20, 2026

View reviewed changes

Comment thread ...polyglot/TypeScript/CommunityToolkit.Aspire.Hosting.K3s/ValidationAppHost/aspire.config.json Outdated

IEvangelist approved these changes May 20, 2026

View reviewed changes

aaronpowell and others added 8 commits May 25, 2026 14:59

Merge branch 'main' into main

90484f4

Merge branch 'main' into main

736b429

chore: providing project reference for polyglot package

9c9e5d5

chore: adding more tests to increase coverage

bd21374

Merge branch 'main' into main

6f90597

fix: adopting with aspire 13.4 changes + improving to docs

c508bc5

Merge branch 'main' into main

b6b03f6

chore: upgrade example app host sdk to match

63f7e02

davidfowl reviewed Jun 2, 2026

View reviewed changes

Comment thread ...mmunityToolkit.Aspire.Hosting.K3s.AppHost/CommunityToolkit.Aspire.Hosting.K3s.AppHost.csproj Outdated

fix: improve api to simplify polyglot sdk

7eaae5f

chore: fixing docs

0673698

edmondshtogu added 4 commits June 8, 2026 13:03

chore: increase aspire version to 13.4.2

6108bda

chore: increase aspire version to 13.4.3

76db209

chore: reverting to 13.4.2 and fixing action to support newer releases

6ff7800

chore: reverting version to 13.4.0

ae9c54b

Merge branch 'CommunityToolkit:main' into main

e99073e

Conversation

edmondshtogu commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of changes

What's included

Key design decisions

Usage

Example

PR Checklist

Other information

Uh oh!

github-actions Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

edmondshtogu commented May 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

IEvangelist left a comment

Choose a reason for hiding this comment

Uh oh!

edmondshtogu commented May 20, 2026

Uh oh!

Uh oh!

davidfowl commented Jun 2, 2026

Uh oh!

edmondshtogu commented Jun 2, 2026

Uh oh!

edmondshtogu commented Jun 5, 2026

Uh oh!

edmondshtogu commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edmondshtogu commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

edmondshtogu commented May 14, 2026 •

edited

Loading

github-actions Bot commented May 14, 2026 •

edited

Loading

edmondshtogu commented Jun 8, 2026 •

edited

Loading