Add 15K node cluster creation pipeline by xinWeiWei24 · Pull Request #1217 · Azure/telescope

xinWeiWei24 · 2026-06-11T10:04:37Z

Adds a KCL pipeline for provisioning a large-scale private AKS cluster, intended for 15K-node performance benchmarking.

Full cluster setup: VNet, subnets, firewall, managed identity, jumpbox, AKS cluster, and user node pools
Jumpbox cloud-init installs Azure CLI, Go, and kubectl
Conditional networkDataplane support — set via network_dataplane parameter, omitted if empty (uses AKS default)
Runtime-configurable parameters (subscription_id, location, cluster_name, user_pool_vm_size, network_dataplane) shown in ADO Run dialog

Pipeline provisions a private AKS cluster with Cilium CNI, Azure Firewall, custom VNet, jumpbox VM, and 14997 user nodes across 19 node pools planned via PlanNodePools. All key params are ADO pipeline variables. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

wonderyl · 2026-06-15T02:59:13Z

+                ),
+                azure.CreateSubnetAndRoleAssignment(
+                    const.DEFAULT_SERVICE_CONNECTION,
+                    RESOURCE_GROUP, VNET_NAME,


nit: one parameter per line please.

wonderyl · 2026-06-15T03:00:23Z

+JUMPBOX_NAME = "${CLUSTER_NAME}-jumpbox"
+
+VNET_CIDR = "10.0.0.0/8"
+NODES_SUBNET_CIDR = "10.1.0.0/18"


The variable is only used once, remove it for better readability.

wonderyl · 2026-06-15T03:15:57Z

+                    const.DEFAULT_SERVICE_CONNECTION,
+                    RESOURCE_GROUP, VNET_NAME,
+                    "nodes", NODES_SUBNET_CIDR,
+                    SUBSCRIPTION_ID, "$(IDENTITY_PRINCIPAL_ID)"


Where is "IDENTITY_PRINCIPAL_ID" defined?

It will be exposed through identity.k.

wonderyl · 2026-06-15T03:18:28Z

+                    VNET_CIDR,
+                    SUBSCRIPTION_ID
+                ),
+                azure.CreateSubnetAndRoleAssignment(


Do you need to assign the role for every subnet? It seems you can set the scope to a resource group, then it grants the role permission to all subnets in the resource group.

wonderyl · 2026-06-15T03:33:19Z

+                    FW_NAME, VNET_NAME, "nodes",
+                    "$(Pipeline.Workspace)/s/kcl/ccp_team/15K_cluster/firewall-policy.json"
+                ),
+                azure.AzCli(


Is it possible to extract this out as a reusable lib?

Do you mean exporting a jumpbox resource library or a regular VM resource library?

jumpbox, I suspect this is not the only pipeline that will need a jumpbox.

wonderyl · 2026-06-15T03:37:29Z

+}
+EOF
+
+az rest \\


I may ask this before, but why use rest api over az aks create?

just follow the old pattern, will use az aks create to hyperscale cluster

This is exactly why we need to scrutinise every line of code, someone or AI may "just follow" the code and pattern.

wonderyl · 2026-06-15T03:51:37Z

+FIREWALL_SUBNET_CIDR = "10.2.0.0/16"
+JUMPBOX_SUBNET_CIDR = "10.3.0.0/16"
+
+# Small values for small scare e2e testing; replace with the lines below for full 15K run


Using 100 contradicts with your PR title and folder name. Please be consistent.

This is just for testing, have tested it and will change it now

wonderyl · 2026-06-15T08:09:01Z

+USER_TOTAL_NODES = 100
+MAX_NODES_PER_POOL = 20
+
+_userPools = [azure.NodePool {


nit, don't need to use underscore variables, nothing is exported.

wonderyl · 2026-06-15T08:22:19Z

+
+VNET_NAME = "${CLUSTER_NAME}-net"
+IDENTITY_NAME = "${CLUSTER_NAME}-identity"
+FW_NAME = "${CLUSTER_NAME}-firewall"


Could you check all the variables and make sure only use a variable when it occurs in multiple places? Using a variable makes the code less readable, so only use it when necessary.

wonderyl · 2026-06-15T08:24:41Z

+  --kubernetes-version "${K8S_VERSION}" \\
+  --dns-name-prefix "${CLUSTER_NAME}-dns" \\
+  --tier standard \\
+  --assign-identity "$(IDENTITY_ID)" \\


where is IDENTITY_ID defined?

Copilot said: I exported it as an environment variable, but it looks less readable. I’ll update the related code to improve readability.

wonderyl · 2026-06-15T08:25:54Z

+  --dns-name-prefix "${CLUSTER_NAME}-dns" \\
+  --tier standard \\
+  --assign-identity "$(IDENTITY_ID)" \\
+  --tags SkipAKSCluster=true \\


I thought we can't use it anymore, no?

This is to skip gatekeeper, not the az sec pack

wonderyl · 2026-06-15T10:08:04Z

+  "Azure Kubernetes Service Cluster Admin Role" \\
+  "Azure Kubernetes Service Cluster User Role" \\
+  "Azure Kubernetes Service Contributor Role" \\


Is AKS contributor role a superset of admin and user role?

❯ az role definition list --name "Azure Kubernetes Service Contributor Role" [ { "assignableScopes": [ "/" ], "createdBy": null, "createdOn": "2020-02-27T19:27:15.073997+00:00", "description": "Grants access to read and write Azure Kubernetes Service clusters", "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/providers/Microsoft.Authorization/roleDefinitions/ed7f3fbd-7b88-4dd4-9017-9adb7ce333f8", "name": "ed7f3fbd-7b88-4dd4-9017-9adb7ce333f8", "permissions": [ { "actions": [ "Microsoft.Authorization/*/read", "Microsoft.ContainerService/locations/*", "Microsoft.ContainerService/managedClusters/*", "Microsoft.ContainerService/managedclustersnapshots/*", "Microsoft.ContainerService/snapshots/*", "Microsoft.Insights/alertRules/*", "Microsoft.Resources/deployments/*", "Microsoft.Resources/subscriptions/resourceGroups/read", "Microsoft.ContainerService/deploymentSafeguards/*" ], "condition": null, "conditionVersion": null, "dataActions": [], "notActions": [], "notDataActions": [] } ], "roleName": "Azure Kubernetes Service Contributor Role", "roleType": "BuiltInRole", "type": "Microsoft.Authorization/roleDefinitions", "updatedBy": null, "updatedOn": "2025-07-24T15:33:31.343706+00:00" } ] ~ via  v22.22.2 ❯ az role definition list --name "Azure Kubernetes Service Cluster User Role" [ { "assignableScopes": [ "/" ], "createdBy": null, "createdOn": "2018-08-15T22:04:53.403724+00:00", "description": "List cluster user credential action.", "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/providers/Microsoft.Authorization/roleDefinitions/4abbcc35-e782-43d8-92c5-2d3f1bd2253f", "name": "4abbcc35-e782-43d8-92c5-2d3f1bd2253f", "permissions": [ { "actions": [ "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action", "Microsoft.ContainerService/managedClusters/read" ], "condition": null, "conditionVersion": null, "dataActions": [], "notActions": [], "notDataActions": [] } ], "roleName": "Azure Kubernetes Service Cluster User Role", "roleType": "BuiltInRole", "type": "Microsoft.Authorization/roleDefinitions", "updatedBy": null, "updatedOn": "2021-11-11T20:13:20.435197+00:00" } ] ~ via  v22.22.2 ❯ az role definition list --name "Azure Kubernetes Service Cluster Admin Role" [ { "assignableScopes": [ "/" ], "createdBy": null, "createdOn": "2018-08-15T21:38:18.595385+00:00", "description": "List cluster admin credential action.", "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/providers/Microsoft.Authorization/roleDefinitions/0ab0b1a8-8aac-4efd-b8c2-3ee1fb270be8", "name": "0ab0b1a8-8aac-4efd-b8c2-3ee1fb270be8", "permissions": [ { "actions": [ "Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action", "Microsoft.ContainerService/managedClusters/accessProfiles/listCredential/action", "Microsoft.ContainerService/managedClusters/read", "Microsoft.ContainerService/managedClusters/runcommand/action" ], "condition": null, "conditionVersion": null, "dataActions": [], "notActions": [], "notDataActions": [] } ], "roleName": "Azure Kubernetes Service Cluster Admin Role", "roleType": "BuiltInRole", "type": "Microsoft.Authorization/roleDefinitions", "updatedBy": null, "updatedOn": "2022-05-17T01:51:12.039065+00:00" } ]

wonderyl · 2026-06-15T10:16:13Z

@@ -0,0 +1,788 @@
+{
+    "location": "eastasia",


In the pipeline, location is a pipeline parameter, here it's a constant, would it cause problems?

Right, will delete it

wonderyl · 2026-06-15T10:19:08Z

+                        },
+                        {
+                            "fqdnTags": [],
+                            "name": "gcr.io",


Is gcr.io a superset of k8s.gcr.io

NO, but *.gcr.io is a superset of k8s.gcr.io, this can be deleted

wonderyl · 2026-06-15T10:20:50Z

+                        },
+                        {
+                            "fqdnTags": [],
+                            "name": "*.gcr.io",


Another gcr.io rule

These 2 FQDN: *.gcr.io and gcr.io are different things in firewall

wonderyl · 2026-06-15T10:23:16Z

+                        },
+                        {
+                            "fqdnTags": [],
+                            "name": "*.lz4.dev",


What is this?

I also forgot the exact purpose, but I remember this FQDN being blocked before and it impacted benchmarking, so I added it to the whitelist.

wonderyl · 2026-06-15T10:25:53Z

+                        },
+                        {
+                            "fqdnTags": [],
+                            "name": "*.quay.io",


is this a superset of quay.io?

*.quay.io and quay.io are different in firewall FQDN setting

wonderyl · 2026-06-15T10:27:55Z

+                        }
+                    ]


Do we actually need to access all these websites, or you copied it from somewhere else?

We need these. I am getting them from 3 different parts:

IMDS endpoint from Azure

endpoints from https://learn.microsoft.com/en-us/azure/aks/outbound-rules-control-egress#azure-global-required-fqdn--application-rules

endpoints behind the CL2 and Prometheus pod images — these can vary each time depending on where the images are served from.

xinWeiWei24 and others added 8 commits June 10, 2026 15:43

update pipeline.k

993e265

update base on the latest v2

1e67f68

changed to parameter

c98965c

changed to parameter

da4c47f

change method to export principal id

e5d6697

change pipeline.k to use principal id to grant role

1948c77

update vm size

10aa65f

xinWeiWei24 requested review from liyu-ma, vittoriasalim and wonderyl as code owners June 11, 2026 10:04

wonderyl reviewed Jun 15, 2026

View reviewed changes

xinWeiWei24 added 4 commits June 15, 2026 15:23

Grant role to vnet

461a87e

set service connection as variable

d10b9cc

set service connection as variable

6465aaf

changed to az cli

15b1265

wonderyl reviewed Jun 15, 2026

View reviewed changes

xinWeiWei24 added 5 commits June 15, 2026 21:30

fix bugs of using control plane flag

53380e5

update to k8s v1.35

42ca966

fix bugs in firewall

7a3f85e

add api server vnet parameter

01fc45c

regenerate jumpbox

0ee39ad

Conversation

xinWeiWei24 commented Jun 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xinWeiWei24 Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xinWeiWei24 Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xinWeiWei24 Jun 15, 2026 •

edited

Loading

xinWeiWei24 Jun 15, 2026 •

edited

Loading