Skip to content

defaultroute0/vks-win-image-builder

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
scripts
scripts
 
 
README.md
README.md
 
 
SESSION_NOTES.md
SESSION_NOTES.md
 
 

Repository files navigation

Complete Beginner's Guide: Building Windows Node Images for vSphere Kubernetes Service

Introduction

This guide walks you through building a Windows Server 2022 node image for vSphere Kubernetes Service (VKS) from scratch. This image allows you to run Windows container workloads on your Kubernetes clusters.

What you'll create: An OVA file containing a customized Windows Server 2022 image with Kubernetes components (kubelet, containerd, etc.) pre-installed.

Prerequisites

Required Infrastructure

  • vCenter Server 8.0 or later
  • DHCP-enabled network (Packer requires DHCP - static IP is not supported)
  • Datastore with at least 50GB free space

Required Software on Build Machine (Ubuntu/Linux)

# Check versions
docker --version    # Required: >= 20.10.21
make --version      # Required: >= 4.2.1
jq --version        # Required: >= 1.6

# Optional but recommended
govc version        # For managing VMs in vCenter

Required Files

  1. Windows Server 2022 ISO - Download from MSDN or Volume Licensing portal
    • Example: en-us_windows_server_2022_updated_april_2022_x64_dvd_d428acee.iso
  2. VMware Tools ISO - Download from Broadcom Knowledge Base
    • Example: VMware-tools-windows-12.5.0-23800621.iso

Understanding Windows Product Keys (IMPORTANT)

What is a KMS Client Key (GVLK)?

When installing Windows unattended, you need a product key. For enterprise/volume licensing environments, Microsoft provides KMS Client Setup Keys (also called GVLK - Generic Volume License Keys).

These are NOT pirated keys - they are officially published by Microsoft for use with Key Management Service (KMS) activation in enterprise environments.

Where Do These Keys Come From?

Microsoft publishes them at: https://learn.microsoft.com/en-us/windows-server/get-started/kms-client-activation-keys

Windows Server 2022 KMS Client Keys

Edition KMS Client Key
Standard VDYBN-27WPP-V4HQT-9VMD4-VMK7H
Datacenter WX4NM-KYWYW-QJJR4-XV3QB-6VM33

How Activation Works

  1. During image build: The KMS key allows Windows to install without manual input
  2. After deployment: Windows nodes activate against your organization's KMS server
  3. For evaluation: Windows runs in evaluation mode if no KMS server is available

Step 1: Clone the Repository

# Clone the VKS Image Builder repository
git clone https://github.com/vmware/vks-image-builder.git
cd vks-image-builder

# Check out the branch for your desired Kubernetes version
git checkout release-1.34   # For Kubernetes 1.34

# Navigate to the image builder directory
cd vsphere-tanzu-kubernetes-grid-image-builder

Step 2: Upload ISOs to vCenter Datastore

You need to upload both ISOs to a datastore accessible by vCenter.

Option A: Using govc (Command Line)

# Set environment variables
export GOVC_URL="your-vcenter.domain.com"
export GOVC_USERNAME="administrator@vsphere.local"
export GOVC_PASSWORD='YourPassword'
export GOVC_INSECURE=1
export GOVC_DATACENTER="YourDatacenter"
export GOVC_DATASTORE="YourDatastore"

# Upload Windows ISO
govc datastore.upload \
  --ds="$GOVC_DATASTORE" \
  --dc="$GOVC_DATACENTER" \
  /path/to/windows_server_2022.iso \
  iso/windows2022.iso

# Upload VMware Tools ISO
govc datastore.upload \
  --ds="$GOVC_DATASTORE" \
  --dc="$GOVC_DATACENTER" \
  /path/to/VMware-tools-windows.iso \
  iso/vmtools.iso

Option B: Using vCenter UI

  1. Log into vCenter web client
  2. Navigate to Storage > Your Datastore > Files
  3. Create an iso folder
  4. Upload both ISO files

Step 3: Configure vSphere Connection

Edit packer-variables/vsphere.j2:

{
    "vcenter_server": "your-vcenter.domain.com",
    "username": "administrator@vsphere.local",
    "password": "YourPassword",
    "datacenter": "YourDatacenter",
    "datastore": "YourDatastore",
    "folder": "",
    "cluster": "YourCluster",
    "network": "YourNetwork",
    "insecure_connection": "true",
    "linked_clone": "true",
    "create_snapshot": "true",
    "destroy": "false"
}

Important: The network MUST have DHCP enabled.

Step 4: Configure Windows ISO Paths

Edit packer-variables/windows/vsphere-windows.j2:

{
  "os_iso_path": "[YourDatastore] iso/windows2022.iso",
  "vmtools_iso_path": "[YourDatastore] iso/vmtools.iso"
}

Note: The format is [DatastoreName] path/to/file.iso with a space after the bracket.

Step 5: Configure the Autounattend File (CRITICAL)

The autounattend.xml file controls the unattended Windows installation. You need to make two critical changes:

Download the Template

curl https://raw.githubusercontent.com/kubernetes-sigs/image-builder/refs/heads/main/images/capi/packer/ova/windows/windows-2022-efi/autounattend.xml \
  -o windows_autounattend.xml

Edit 1: Add Product Key

Find the <UserData> section and add the <ProductKey> element:

<UserData>
    <AcceptEula>true</AcceptEula>
    <FullName>Administrator</FullName>
    <Organization>Organization</Organization>
    <ProductKey>
        <Key>VDYBN-27WPP-V4HQT-9VMD4-VMK7H</Key>
    </ProductKey>
</UserData>

Edit 2: Set Windows Edition (Image Index)

Find the <InstallFrom> section and set the image index:

Before (won't work):

<Value>{{user `windows_image_index`}}</Value>

After (hardcoded):

<Value>2</Value>

Windows Server 2022 Image Indices

Index Edition
1 Standard (Server Core - no GUI)
2 Standard (Desktop Experience - with GUI)
3 Datacenter (Server Core - no GUI)
4 Datacenter (Desktop Experience - with GUI)

Recommendation: Use index 2 for Standard with Desktop Experience.

Step 6: Build the Image

6.1 Clean Up Any Previous Build Artifacts

# Remove old containers
make clean-containers

6.2 Start the Artifacts Container

make run-artifacts-container ARTIFACTS_CONTAINER_PORT=8081

6.3 Verify Artifacts Container is Running

# Check container is running
docker ps --filter "label=byoi_artifacts"

# Test the endpoint (IMPORTANT - do this before building!)
curl -s http://127.0.0.1:8081/artifacts/metadata/kubernetes_config.json | head -5

Expected output:

{
  "kubernetes": "v1.34.1+vmware.1",
  "coredns": "v1.12.1+vmware.5-fips",
  ...
}

If this fails, DO NOT proceed to the build step!

6.4 Start the Build

make build-node-image \
  OS_TARGET=windows-2022-efi \
  TKR_SUFFIX=vkr.4 \
  HOST_IP=<YOUR_BUILD_MACHINE_IP> \
  IMAGE_ARTIFACTS_PATH=$(pwd)/image-artifacts \
  ARTIFACTS_CONTAINER_PORT=8081 \
  AUTO_UNATTEND_ANSWER_FILE_PATH=$(pwd)/windows_autounattend.xml

Replace <YOUR_BUILD_MACHINE_IP> with the IP address of your build machine that is reachable from vCenter.

Step 7: Monitor the Build

Watch Logs in Real-Time

docker logs -f v1.34.1---vmware.1-windows-2022-efi-image-builder

Check Recent Log Output

docker logs v1.34.1---vmware.1-windows-2022-efi-image-builder 2>&1 | tail -50

Build Stages

Stage Description Duration
Creating virtual machine VM created in vCenter ~1 min
Waiting for IP Windows installing from ISO 10-20 min
WinRM connected Packer connected to Windows -
Provisioning with Ansible Installing K8s components 15-30 min
Exporting virtual machine Creating OVA file 5-10 min

Total build time: 30-60 minutes

Step 8: Verify Success

Check for OVA File

ls -la image-artifacts/ovas/

You should see a file like:

windows-2022-efi-kube-v1.34.1-vkr.4-20260121.ova

Check Build Logs

ls -la image-artifacts/logs/

Troubleshooting Guide

Problem: Build Exits with Code 4

Symptom:

docker ps -a
# Shows container with "Exited (4)" status

Cause: Artifacts container was not running.

Solution:

  1. Start artifacts container first: make run-artifacts-container
  2. Verify it's accessible: curl http://127.0.0.1:8081/artifacts/metadata/kubernetes_config.json
  3. Then start the build

Problem: "Windows cannot read the product key"

Symptom: vCenter console shows this error during Windows install.

Cause: Missing or invalid ProductKey in autounattend.xml.

Solution: Add the ProductKey section to your autounattend.xml:

<ProductKey>
    <Key>VDYBN-27WPP-V4HQT-9VMD4-VMK7H</Key>
</ProductKey>

Problem: Windows Shows "Select Operating System" Screen

Symptom: Instead of automatic install, Windows asks you to choose an edition.

Cause: The image index is not being set correctly (Packer variable not replaced).

Solution: Hardcode the image index in autounattend.xml:

<Value>2</Value>

Problem: Build Stuck at "Waiting for IP"

Symptom: Logs show Waiting for IP... for more than 20 minutes.

Causes & Solutions:

  1. Check vCenter console - Look for Windows installation errors
  2. DHCP not working - Verify the VM's network has DHCP
  3. ISO mount issues - Verify ISO paths are correct in vsphere-windows.j2

Problem: WinRM Timeout Errors

Symptom:

ERROR DURING WINRM SEND INPUT - attempting to recover: ReadTimeout

Cause: Normal - Windows is slow to respond, Ansible will retry.

Solution: Wait - these are usually recoverable. Only worry if the build completely fails.

Managing VMs with govc

Setup

export GOVC_URL="your-vcenter.domain.com"
export GOVC_USERNAME="administrator@vsphere.local"
export GOVC_PASSWORD='YourPassword'
export GOVC_INSECURE=1
export GOVC_DATACENTER="YourDatacenter"

Find Build VMs

govc find . -type m -name "windows-2022-efi-kube*"

Delete Failed VMs

govc vm.power -off -force "./vm/windows-2022-efi-kube-v1.34.1-TIMESTAMP"
govc vm.destroy "./vm/windows-2022-efi-kube-v1.34.1-TIMESTAMP"

Post-Build: Deploying the OVA

1. Upload to Content Library

  1. Download OVA from image-artifacts/ovas/
  2. In vCenter, go to Content Libraries
  3. Create or select a local library
  4. Upload the OVA as a new item
  5. Important: Disable security policies for Windows images

2. Create Kubernetes Cluster

  • Use vSphere Kubernetes Service 3.4+ documentation
  • Windows nodes require best-effort-large or larger VM class
  • You need both Linux (for control plane) and Windows node images

Quick Reference

Key Files

File Purpose
packer-variables/vsphere.j2 vCenter connection
packer-variables/windows/vsphere-windows.j2 ISO paths
packer-variables/windows/default-args-windows.j2 Build defaults
windows_autounattend.xml Windows installation settings

Key Commands

# Clean up
make clean-containers

# Start artifacts
make run-artifacts-container ARTIFACTS_CONTAINER_PORT=8081

# Build
make build-node-image OS_TARGET=windows-2022-efi ...

# Monitor
docker logs -f v1.34.1---vmware.1-windows-2022-efi-image-builder

# Check output
ls image-artifacts/ovas/

Windows KMS Keys (Official Microsoft)

Edition Key
Server 2022 Standard VDYBN-27WPP-V4HQT-9VMD4-VMK7H
Server 2022 Datacenter WX4NM-KYWYW-QJJR4-XV3QB-6VM33

Source: https://learn.microsoft.com/en-us/windows-server/get-started/kms-client-activation-keys

References

Appendix A: Exact Configuration Files Used

This section shows the exact configuration files that were modified from the default repository.

File 1: packer-variables/vsphere.j2

{
    "vcenter_server":"vc-wld01-a.site-a.vcf.lab",
    "username":"administrator@wld.sso",
    "password":"VMware123!VMware123!",
    "datacenter":"wld-01a-DC",
    "datastore":"cluster-wld01-01a-vsan01",
    "folder":"",
    "cluster": "cluster-wld01-01a",
    "network": "ryantest-public",
    "insecure_connection": "true",
    "linked_clone": "true",
    "create_snapshot": "true",
    "destroy": "false"
}

File 2: packer-variables/windows/vsphere-windows.j2

{
  "os_iso_path": "[cluster-wld01-01a-vsan01] iso/en-us_windows_server_2022_updated_april_2022_x64_dvd_d428acee.iso",
  "vmtools_iso_path": "[cluster-wld01-01a-vsan01] iso/windows.iso"
}

File 3: packer-variables/windows/default-args-windows.j2

Key changes:

  • Added memory and cpu settings (8GB RAM, 4 CPU)
  • Added WinRM timeout settings in ansible_user_vars
  • Added windows_product_key
{
  "memory": "8192",
  "cpu": "4",
  "additional_executables_destination_path": "C:\\ProgramData\\Temp",
  "additional_executables_list": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/registry.exe,http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/goss.exe",
  "additional_executables": "true",
  "additional_url_images": "false",
  "additional_url_images_list": "",
  "additional_prepull_images": "",
  "build_version": "{{ os_type }}-kube-{{ kubernetes_series }}-{{ ova_ts_suffix }}",
  "cloudbase_init_url": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/CloudbaseInitSetup_x64.msi",
  "cloudbase_real_time_clock_utc": "true",
  "containerd_url": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/cri-containerd.tar",
  "containerd_sha256_windows": "{{ containerd_sha256_windows_amd64 }}",
  "containerd_version": "{{ containerd }}",
  "convert_to_template": "true",
  "create_snapshot": "false",
  "disable_hypervisor": "false",
  "disk_size": "40960",
  "kubernetes_base_url": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64",
  "kubernetes_series": "{{ kubernetes_series }}",
  "kubernetes_semver": "{{ kubernetes_version }}",
  "kubernetes_typed_version": "{{ image_version }}",
  "load_additional_components": "true",
  "netbios_host_name_compatibility": "false",
  "nssm_url": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/nssm.exe",
  "prepull": "false",
  "pause_image": "localhost:5000/vmware.io/pause:{{ pause }}",
  "runtime": "containerd",
  "template": "",
  "unattend_timezone": "Pacific Standard Time",
  "windows_updates_categories": "",
  "windows_updates_kbs": "",
  "wins_url": "",
  "custom_role": "true",
  "custom_role_names": "/image-builder/images/capi/image/ansible-windows",
  "ansible_user_vars": "ansible_winrm_operation_timeout_sec=120 ansible_winrm_read_timeout_sec=150 artifacts_container_url=http://{{ host_ip }}:{{ artifacts_container_port }} imageVersion={{ image_version|replace('-', '.') }} registry_store_archive_url=http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/registries/{{ registry_store_path }}",
  "vmx_version": "21",
  "debug_tools": "false",
  "enable_auto_kubelet_service_restart": "false",
  "windows_admin_password": "VMware123!VMware123!",
  "windows_image_index": "3",
  "windows_product_key": "VDYBN-27WPP-V4HQT-9VMD4-VMK7H"
}

File 4: windows_autounattend.xml (Critical Sections)

ProductKey section (added):

<UserData>
    <AcceptEula>true</AcceptEula>
    <FullName>Administrator</FullName>
    <Organization>Organization</Organization>
    <ProductKey>
        <Key>VDYBN-27WPP-V4HQT-9VMD4-VMK7H</Key>
    </ProductKey>
</UserData>

Image Index section (hardcoded to 2):

<InstallFrom>
    <MetaData wcm:action="add">
        <Key>/IMAGE/INDEX</Key>
        <Value>2</Value>
    </MetaData>
</InstallFrom>

Appendix B: Verification Commands

Before Starting Build

# 1. Verify Docker is running
docker info > /dev/null 2>&1 && echo "Docker OK" || echo "Docker NOT running"

# 2. Check required tools
docker --version
make --version
jq --version

# 3. Verify vCenter connectivity (requires govc)
export GOVC_URL="your-vcenter"
export GOVC_USERNAME="user@domain"
export GOVC_PASSWORD='password'
export GOVC_INSECURE=1
govc about

# 4. Verify ISOs exist on datastore
govc datastore.ls -ds=YourDatastore iso/

# 5. Verify artifacts container is running
docker ps --filter "label=byoi_artifacts"

# 6. Test artifacts endpoint
curl -s http://127.0.0.1:8081/artifacts/metadata/kubernetes_config.json | jq .kubernetes

# 7. Verify ports are available/correct
ss -tlnp | grep -E '8081|8082'

During Build

# Check container status
docker ps --filter "name=windows-2022-efi"

# View live logs
docker logs -f v1.34.1---vmware.1-windows-2022-efi-image-builder

# Check specific Ansible tasks
docker logs v1.34.1---vmware.1-windows-2022-efi-image-builder 2>&1 | grep "TASK \["

# Check for errors
docker logs v1.34.1---vmware.1-windows-2022-efi-image-builder 2>&1 | grep -i "error\|failed"

# Find the VM in vCenter
govc find . -type m -name "windows-2022-efi-kube*"

# Check VM power state
govc vm.info "windows-2022-efi-kube-v1.34.1-TIMESTAMP"

After Build

# Check for OVA output
ls -la image-artifacts/ovas/

# Check OVA file size (should be several GB)
du -h image-artifacts/ovas/*.ova

# Check build logs
ls -la image-artifacts/logs/
cat image-artifacts/logs/packer-*.log | tail -100

# Verify container exited successfully (exit code 0)
docker ps -a --filter "name=windows-2022-efi" --format "{{.Names}}: {{.Status}}"

Appendix C: Summary of Changes from Default Repository

File Change Why
packer-variables/vsphere.j2 Updated vCenter connection details Required for your environment
packer-variables/windows/vsphere-windows.j2 Set ISO paths Point to your uploaded ISOs
packer-variables/windows/default-args-windows.j2 Added windows_product_key Required for unattended Windows install
windows_autounattend.xml Added <ProductKey> element Required for unattended Windows install
windows_autounattend.xml Changed image index from {{user...}} to 2 Packer variable wasn't being replaced

Appendix D: Disk Space Management

Minimum Space Requirements

  • During build: ~15GB free recommended
  • OVA export: The VMDK is ~26GB uncompressed, OVA is ~5-6GB compressed
  • Watch disk usage during export phase - this is when it fails most often

Pre-Build Cleanup Commands

# Check current disk space
df -h /

# Clean Docker (if not using build containers)
docker system prune -f

# Clean apt cache
sudo apt clean

# Trim journal logs
sudo journalctl --vacuum-size=50M

# Remove old snap versions (needs sudo)
sudo snap set system refresh.retain=2
snap list --all | awk '/disabled/{system("sudo snap remove " $1 " --revision=" $3)}'

If Build Fails During Export (Low Space)

If the build completes but export fails due to "no space left on device":

  1. The template still exists in vCenter (if destroy: false is set)
  2. Export manually from vCenter UI:
    • Right-click the template → Export OVF Template
    • Save to local machine
  3. Convert OVF to OVA using ovftool:
cd /path/to/exported/files
ovftool template-name.ovf template-name.ova

Clean Orphaned Build Files on vSAN

export GOVC_URL="vc-wld01-a.site-a.vcf.lab"
export GOVC_USERNAME="administrator@wld.sso"
export GOVC_PASSWORD="VMware123!VMware123!"
export GOVC_INSECURE=true

# List orphaned build folders
govc datastore.ls -ds=cluster-wld01-01a-vsan01 | grep windows-2022-efi-kube

# Delete them (run for each folder)
govc datastore.rm -ds=cluster-wld01-01a-vsan01 "folder-name"

Appendix E: Deploying Windows Node Pools to VKS

Prerequisites for Windows Node Pools

  1. Kubernetes version v1.31+ - Windows not supported on earlier versions
  2. ClusterClass builtin-generic-v3.2.0+ - Required for Windows
  3. VM Class: best-effort-large or larger - Windows needs more resources
  4. Content Library with Windows OVA uploaded
  5. Antrea CNI - Required for Windows networking

Upload OVA to Content Library

  1. In vCenter, navigate to Content Libraries
  2. Create or select a local library (e.g., "win-ova")
  3. Click "Actions" → "Import Item"
  4. Upload the OVA file
  5. Associate the content library with your namespace

Create New Cluster with Windows Support

See windows-cluster-ns01.yaml in mydocs folder:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: win-cluster01
  namespace: ns01
spec:
  topology:
    class: builtin-generic-v3.2.0
    version: v1.34.1+vmware.1-vkr.4
    controlPlane:
      replicas: 1
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: os-name=photon
    workers:
      machineDeployments:
        - class: node-pool
          name: linux-pool
          replicas: 1
          metadata:
            annotations:
              run.tanzu.vmware.com/resolve-os-image: os-name=photon
        - class: node-pool
          name: windows-pool
          replicas: 1
          metadata:
            annotations:
              run.tanzu.vmware.com/resolve-os-image: os-type=windows
          variables:
            overrides:
              - name: vmClass
                value: best-effort-large
    variables:
      - name: vmClass
        value: best-effort-small
      - name: storageClass
        value: vsan-default-storage-policy

Add Windows Node Pool to Existing Cluster

Note: Cluster must be v1.31+ with ClusterClass v3.2.0+

kubectl patch cluster guest-cluster01 -n ns01 --type=json -p='[
  {
    "op": "add",
    "path": "/spec/topology/workers/machineDeployments/-",
    "value": {
      "class": "node-pool",
      "name": "windows-pool",
      "replicas": 1,
      "metadata": {
        "annotations": {
          "run.tanzu.vmware.com/resolve-os-image": "os-type=windows"
        }
      },
      "variables": {
        "overrides": [
          {"name": "vmClass", "value": "best-effort-large"}
        ]
      }
    }
  }
]'

Content Library Disambiguation

When multiple content libraries are associated with a namespace:

metadata:
  annotations:
    # Specify by name
    run.tanzu.vmware.com/resolve-os-image: os-type=windows,content-library=win-ova

    # Or by ID
    run.tanzu.vmware.com/resolve-os-image: os-type=windows,content-library=cl-xxxxx

Verify Windows Node Pool

# Check cluster status
kubectl get cluster -n ns01

# Check machine deployments
kubectl get machinedeployment -n ns01

# Check machines
kubectl get machine -n ns01

# Check Windows node joined
kubectl get nodes -l kubernetes.io/os=windows

Appendix F: Files in mydocs/ Folder

File Description
vsphere.j2 vCenter connection config (destroy: false)
vsphere-windows.j2 Windows ISO paths on datastore
default-args-windows.j2 Build args (8GB RAM, WinRM timeouts, product key)
windows_autounattend.xml Autounattend with ProductKey and image index
windows-cluster-ns01.yaml YAML for new cluster with Windows support
windows-nodepool-patch.yaml Patch for adding Windows nodepool
SESSION_NOTES.md Session notes and lessons learned
WINDOWS_BUILD_GUIDE.md This guide

Guide created and tested during successful Windows node image build sessions - January 2026

About

manually building a windows nodepool

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors