From e50907afb6e10f64cfb3df352f61bbc16f66501b Mon Sep 17 00:00:00 2001 From: Melvin Hillsman Date: Wed, 10 Jun 2026 21:29:15 -0500 Subject: [PATCH 1/4] chore: remove frozen historical snapshot docs Signed-off-by: Melvin Hillsman --- docs/implementation-plan.md | 1049 ----------------- ...hift-virtualization-workload-automation.md | 277 ----- 2 files changed, 1326 deletions(-) delete mode 100644 docs/implementation-plan.md delete mode 100644 docs/openshift-virtualization-workload-automation.md diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md deleted file mode 100644 index be00b40..0000000 --- a/docs/implementation-plan.md +++ /dev/null @@ -1,1049 +0,0 @@ -# Virtwork Implementation Plan - -> **Historical snapshot — original phased build (phases 0–12).** -> -> This document describes the original incremental plan executed to build virtwork through phase 12. It is preserved as a record of the layered design and as context for why packages are organized the way they are. **Subsequent additions are not reflected here.** For the current state, see: -> -> - Chaos workloads (chaos-disk, chaos-network, chaos-process) → [chaos-workloads.md](chaos-workloads.md) -> - TPS workload → [architecture.md](architecture.md) and [guide/02-deploying-workloads.md](guide/02-deploying-workloads.md) -> - Structured logging (`internal/logging`) → [architecture.md](architecture.md) and [development.md](development.md) -> - Golden container disk image → [../build/golden-image/README.md](../build/golden-image/README.md) -> - Audit schema (5 tables) → [audit-schema.md](audit-schema.md) -> - Complete configuration reference → [configuration.md](configuration.md) -> - OpenShift deployment deep-dive → [deployment.md](deployment.md) - -This document breaks the [design plan](openshift-virtualization-workload-automation.md) into incremental phases. Each phase produces a testable, committable increment that builds on the previous one. See [architecture.md](architecture.md) for the high-level architecture and diagrams. - ---- - -## Phase Summary - -| Phase | Layer | Packages | Tests (approx) | Goroutines? | -|-------|-------|----------|-----------------|-------------| -| 0 | — | Project scaffold, go.mod, Ginkgo bootstrap | 1 smoke | No | -| 1 | 0+1 | `internal/constants`, `internal/config` | 14-16 | No | -| 2 | 1 | `internal/cloudinit` | 18-20 | No | -| 3 | 1 | `internal/cluster` | 5 | No | -| 4 | 2 | `internal/vm` | 18-20 | Yes (CRUD) | -| 5 | 2 | `internal/resources`, `internal/wait` | 11 | Yes | -| 6 | 3 | `internal/workloads` (interface, cpu, memory, disk) | 22-26 | No | -| 7 | 3 | `internal/workloads` (database, network) | 14-16 | No | -| 8 | 3 | `internal/workloads` (registry) | 10 | No | -| 9 | 4 | `internal/cleanup` | 7-10 | Yes | -| 10 | 4 | `cmd/virtwork` (Cobra CLI + orchestration) | 22-25 + BDD | Yes | -| 11 | — | Integration tests (alongside source) + E2E tests (`tests/e2e/`) | ~51 | Yes | -| 12 | 4 | `internal/audit` (SQLite audit tracking) | 14 | No | -| **Total** | | **12 packages + testutil + e2e** | **~220 tests** | | - -**Note:** Test counts increased from the initial estimate to account for SSH credential support (cloud-init users block, config fields, CLI flags) and the memory workload. These are lessons from the Python implementation experience that revealed the true scope. - ---- - -## Phase 0: Project Foundation - -**Goal:** Establish the project skeleton so that `go test ./...` passes and the Ginkgo test suite is bootstrapped. - -### Files to Create - -- `cmd/virtwork/main.go` — minimal `func main()` placeholder -- `internal/constants/constants.go` — package declaration only -- `internal/config/config.go` — package declaration only -- `internal/cluster/cluster.go` — package declaration only -- `internal/cloudinit/cloudinit.go` — package declaration only -- `internal/vm/vm.go` — package declaration only -- `internal/resources/resources.go` — package declaration only -- `internal/wait/wait.go` — package declaration only -- `internal/cleanup/cleanup.go` — package declaration only -- `internal/workloads/workload.go` — package declaration only - -### Ginkgo Bootstrap - -Bootstrap Ginkgo test suites for each package: - -```bash -cd internal/constants && ginkgo bootstrap -cd internal/config && ginkgo bootstrap -cd internal/cloudinit && ginkgo bootstrap -cd internal/cluster && ginkgo bootstrap -cd internal/vm && ginkgo bootstrap -cd internal/resources && ginkgo bootstrap -cd internal/wait && ginkgo bootstrap -cd internal/cleanup && ginkgo bootstrap -cd internal/workloads && ginkgo bootstrap -``` - -Create a smoke test: - -- `internal/constants/constants_test.go` — one `Describe` block asserting the package is importable - -### Files to Modify - -- `go.mod`: - - Add required dependencies: `github.com/spf13/cobra`, `github.com/spf13/viper`, `gopkg.in/yaml.v3` - - Add K8s dependencies: `sigs.k8s.io/controller-runtime`, `k8s.io/api`, `k8s.io/apimachinery`, `k8s.io/client-go` - - Add KubeVirt dependencies: `kubevirt.io/api`, `kubevirt.io/containerized-data-importer-api` - - Add test dependencies: `github.com/onsi/ginkgo/v2`, `github.com/onsi/gomega` - -### Verification - -- `go build ./...` succeeds -- `go test ./...` → 1 test collected, 1 passed -- `ginkgo -r` runs the smoke test - -### Commits - -- `chore: scaffold virtwork package structure and Ginkgo test infrastructure` - ---- - -## Phase 1: Constants and Config - -**Goal:** Define all project constants and the Viper-based configuration system. Pure Go — no external system interaction. - -### Files to Create - -**`internal/constants/constants.go`** -- KubeVirt API: `KubevirtAPIGroup`, `KubevirtAPIVersion`, `KubevirtVMPlural`, `KubevirtVMIPlural` -- CDI API: `CDIAPIGroup`, `CDIAPIVersion`, `CDIDVPlural` -- Defaults: `DefaultContainerDiskImage` (quay.io/containerdisks/fedora:41), `DefaultNamespace` (virtwork), `DefaultCPUCores` (2), `DefaultMemory` (2Gi), `DefaultDiskSize` (10Gi), `DefaultSSHUser` (virtwork) -- Labels: `LabelAppName`, `LabelManagedBy`, `LabelComponent`, `ManagedByValue` -- Polling: `DefaultReadyTimeout` (600 * time.Second), `DefaultPollInterval` (15 * time.Second) - -**`internal/config/config.go`** -- `WorkloadConfig` struct: `Enabled bool`, `VMCount int`, `CPUCores int`, `Memory string` -- `Config` struct: `Namespace`, `ContainerDiskImage`, `DataDiskSize`, `Workloads map[string]WorkloadConfig`, `KubeconfigPath`, `CleanupMode`, `WaitForReady`, `ReadyTimeoutSeconds`, `DryRun`, `Verbose`, `SSHUser`, `SSHPassword`, `SSHAuthorizedKeys` -- List fields (`SSHAuthorizedKeys`) need explicit handling outside the generic Viper binding loop: YAML passes lists directly, env vars split on commas, CLI merges from `--ssh-key` and `--ssh-key-file` -- `LoadConfig(cmd *cobra.Command) (*Config, error)` — reads from Viper (Cobra flags > env > YAML > defaults) -- `SetDefaults()` — registers Viper defaults -- `BindFlags(cmd *cobra.Command)` — binds Cobra flags to Viper keys - -### Tests (write first — TDD) - -**`internal/constants/constants_test.go`** (Ginkgo): -```go -Describe("Constants", func() { - It("should have correct KubeVirt API group", ...) - It("should have correct default namespace", ...) - It("should have valid label key format", ...) -}) -``` - -**`internal/config/config_test.go`** (Ginkgo): -```go -Describe("Config", func() { - Context("with defaults", func() { - It("should have correct default namespace", ...) - It("should have correct default CPU cores", ...) - }) - Context("with YAML config file", func() { - It("should load namespace from file", ...) - It("should return error for invalid YAML", ...) - It("should return error for missing file", ...) - }) - Context("with environment variables", func() { - It("should override defaults with VIRTWORK_ env vars", ...) - }) - Context("priority chain", func() { - It("should prefer flags over env vars", ...) - It("should prefer env vars over config file", ...) - }) - Context("SSH config fields", func() { - It("should default SSHUser to virtwork", ...) - It("should default SSHPassword to empty", ...) - It("should default SSHAuthorizedKeys to empty", ...) - It("should split comma-separated VIRTWORK_SSH_AUTHORIZED_KEYS", ...) - }) -}) -``` - -### Verification - -- `go test ./internal/constants/... ./internal/config/...` all pass -- `ginkgo -r internal/constants internal/config` all pass - -### Commits - -- `feat: add constants package with KubeVirt API coordinates and defaults` -- `feat: add config package with Viper-based priority chain` - ---- - -## Phase 2: Cloud-init Builder - -**Goal:** Build the `cloudinit` package — pure string/YAML manipulation, no K8s dependency. - -### Files to Create - -**`internal/cloudinit/cloudinit.go`** -- `CloudConfigOpts` struct: `Packages []string`, `WriteFiles []WriteFile`, `RunCmd [][]string`, `Extra map[string]interface{}`, `SSHUser string`, `SSHPassword string`, `SSHAuthorizedKeys []string` -- `WriteFile` struct: `Path string`, `Content string`, `Permissions string` -- `BuildCloudConfig(opts CloudConfigOpts) (string, error)` -- Returns `#cloud-config\n` + YAML marshal -- Omits keys with empty/nil values -- When `SSHUser` is non-empty, emits a `users` block: - - `name`, `sudo: ALL=(ALL) NOPASSWD:ALL`, `shell: /bin/bash` - - `lock_passwd: true` when no password (key-only auth), `lock_passwd: false` + `plain_text_passwd` when password set - - `ssh_authorized_keys` list when keys provided - - `ssh_pwauth: true` at top level only when password is set -- No `users` block when `SSHUser` is empty (backward compatible) - -### Tests (TDD) - -**`internal/cloudinit/cloudinit_test.go`** (Ginkgo): -```go -Describe("BuildCloudConfig", func() { - It("should return valid #cloud-config header for empty opts", ...) - It("should include packages list", ...) - It("should include write_files with correct structure", ...) - It("should include runcmd entries", ...) - It("should merge extra keys at top level", ...) - It("should omit nil/empty values", ...) - It("should produce output parseable by yaml.Unmarshal", ...) - It("should have exact #cloud-config header", ...) - - Context("SSH user support", func() { - It("should create users block when SSHUser is set", ...) - It("should set lock_passwd true when no password", ...) - It("should set lock_passwd false and plain_text_passwd when password set", ...) - It("should set ssh_pwauth true only when password is set", ...) - It("should include ssh_authorized_keys when keys provided", ...) - It("should handle multiple authorized keys", ...) - It("should handle combined password and keys", ...) - It("should not create users block when SSHUser is empty", ...) - It("should not include ssh_authorized_keys when keys list is empty", ...) - It("should coexist with workload params (packages, runcmd)", ...) - }) -}) -``` - -**Implementation lesson from Python:** When testing YAML output, always parse the YAML string with `yaml.Unmarshal` before asserting on values — never assert on raw string content. YAML serializers may reorder keys, fold long lines (Go's `gopkg.in/yaml.v3` handles this differently than Python's PyYAML, but the principle holds), or vary whitespace. - -### Verification - -- `go test ./internal/cloudinit/...` all pass -- Output round-trips through `yaml.Unmarshal` - -### Commits - -- `feat: add cloud-init YAML builder` - ---- - -## Phase 3: Cluster Client - -**Goal:** controller-runtime client initialization — first package with external system interaction. - -### Files to Create - -**`internal/cluster/cluster.go`** -- `Connect(kubeconfigPath string) (client.Client, error)` - - Build runtime scheme with core types, KubeVirt types, CDI types - - Try `rest.InClusterConfig()` - - On error, fall back to `clientcmd.BuildConfigFromFlags("", kubeconfigPath)` - - Both fail → return wrapped error - - Create `client.New(restConfig, client.Options{Scheme: scheme})` -- `NewScheme() *runtime.Scheme` — registers all needed types - -### Tests (TDD) - -**`internal/cluster/cluster_test.go`** (Ginkgo): -```go -Describe("Connect", func() { - It("should return error when both in-cluster and kubeconfig fail", ...) - It("should use kubeconfig path when provided", ...) -}) -Describe("NewScheme", func() { - It("should register core types", ...) - It("should register KubeVirt types", ...) - It("should register CDI types", ...) -}) -``` - -### Verification - -- `go test ./internal/cluster/...` all pass -- `Connect()` returns `(client.Client, error)` - -### Commits - -- `feat: add cluster client initialization with controller-runtime and scheme registration` - ---- - -## Phase 4: VM Spec Building and CRUD - -**Goal:** VM spec construction (pure functions) + K8s CRUD operations with retry logic. Largest single package. - -### Files to Create - -**`internal/vm/vm.go`** - -Types: -- `VMSpecOpts` struct: `Name`, `Namespace`, `ContainerDiskImage`, `CloudInitUserdata`, `CPUCores`, `Memory`, `Labels`, `ExtraDisks`, `ExtraVolumes`, `DataVolumeTemplates` - -Pure functions: -- `BuildVMSpec(opts VMSpecOpts) *kubevirtv1.VirtualMachine` - - `containerDisk` for OS, `cloudInitNoCloud` for userdata, masquerade networking, virtio bus, `spec.Running: true` -- `BuildDataVolumeTemplate(name, size string) cdiv1beta1.DataVolumeTemplateSpec` - -CRUD operations: -- `CreateVM(ctx context.Context, c client.Client, vm *kubevirtv1.VirtualMachine) error` — AlreadyExists=skip, transient=retry -- `DeleteVM(ctx context.Context, c client.Client, name, namespace string) error` -- `ListVMs(ctx context.Context, c client.Client, namespace string, labels map[string]string) ([]kubevirtv1.VirtualMachine, error)` -- `GetVMIPhase(ctx context.Context, c client.Client, name, namespace string) (kubevirtv1.VirtualMachineInstancePhase, error)` -- `retryOnTransient(ctx context.Context, fn func() error, maxRetries int) error` — unexported retry helper - -### Tests (TDD) - -**`internal/vm/vm_test.go`** (Ginkgo): -```go -Describe("BuildVMSpec", func() { - It("should set correct API version and kind", ...) - It("should set name and namespace", ...) - It("should configure containerDisk volume", ...) - It("should configure cloudInitNoCloud volume", ...) - It("should set labels", ...) - It("should set CPU and memory resources", ...) - It("should set running to true", ...) - It("should configure masquerade networking", ...) - It("should include extra disks when provided", ...) - It("should include data volume templates when provided", ...) -}) - -Describe("CreateVM", func() { - It("should create VM successfully", ...) - It("should skip on AlreadyExists", ...) - It("should retry on transient errors", ...) - It("should fail on NotFound", ...) - It("should fail on Unauthorized", ...) -}) - -Describe("DeleteVM", func() { ... }) -Describe("ListVMs", func() { ... }) -Describe("GetVMIPhase", func() { ... }) -``` - -### Verification - -- `go test ./internal/vm/...` all pass -- `BuildVMSpec()` output matches KubeVirt VirtualMachine schema -- Retry logic covers all documented error conditions - -### Commits - -- `feat: add VM spec builder with container disk and cloud-init support` -- `feat: add VM CRUD operations with retry logic` - ---- - -## Phase 5: Resources and Wait - -**Goal:** Complete Layer 2 — namespace/service helpers and VMI readiness polling. - -### Files to Create - -**`internal/resources/resources.go`** -- `EnsureNamespace(ctx context.Context, c client.Client, name string, labels map[string]string) error` — create if not exists, AlreadyExists=skip -- `CreateService(ctx context.Context, c client.Client, svc *corev1.Service) error` — AlreadyExists=skip -- `DeleteManagedServices(ctx context.Context, c client.Client, namespace string, labels map[string]string) (int, error)` — returns count - -**`internal/wait/wait.go`** -- `WaitForVMReady(ctx context.Context, c client.Client, name, namespace string, timeout, interval time.Duration) error` — polls VMI phase, uses `time.Sleep()` -- `WaitForAllVMsReady(ctx context.Context, c client.Client, names []string, namespace string, timeout, interval time.Duration) map[string]error` — `errgroup.Group` for concurrent polling - -### Tests (TDD) - -**`internal/resources/resources_test.go`** (Ginkgo): -```go -Describe("EnsureNamespace", func() { - It("should create namespace", ...) - It("should skip on AlreadyExists", ...) - It("should apply labels", ...) -}) -Describe("CreateService", func() { - It("should create service", ...) - It("should skip on AlreadyExists", ...) -}) -Describe("DeleteManagedServices", func() { - It("should return delete count", ...) -}) -``` - -**`internal/wait/wait_test.go`** (Ginkgo): -```go -Describe("WaitForVMReady", func() { - It("should return nil when immediately ready", ...) - It("should return nil after N polls", ...) - It("should return error on timeout", ...) -}) -Describe("WaitForAllVMsReady", func() { - It("should poll all VMs concurrently", ...) - It("should report partial failures", ...) -}) -``` - -### Verification - -- `go test ./internal/resources/... ./internal/wait/...` all pass -- `WaitForVMReady` uses `time.Sleep()` not blocking the caller indefinitely -- `WaitForAllVMsReady` uses `errgroup.Group` - -### Commits - -- `feat: add namespace and service resource helpers` -- `feat: add VMI readiness polling with concurrent support` - ---- - -## Phase 6: Workload Interface + CPU + Memory + Disk - -**Goal:** Define the Workload interface and implement the three simpler workloads. These are pure data producers — no I/O, no goroutines. - -### Files to Create - -**`internal/workloads/workload.go`** -- `Workload` interface: `Name()`, `CloudInitUserdata()`, `VMResources()`, `ExtraVolumes()`, `ExtraDisks()`, `DataVolumeTemplates()`, `RequiresService()`, `ServiceSpec()`, `VMCount()` -- `VMResourceSpec` struct: `CPUCores int`, `Memory string` -- `BaseWorkload` struct: `Config WorkloadConfig`, `SSHUser string`, `SSHPassword string`, `SSHAuthorizedKeys []string` — embedded struct providing defaults for optional methods - - `ExtraVolumes() → nil`, `ExtraDisks() → nil`, `DataVolumeTemplates() → nil`, `RequiresService() → false`, `ServiceSpec() → nil`, `VMCount() → 1` - - `BuildCloudConfig(opts CloudConfigOpts) (string, error)` — helper that injects SSH credentials into `cloudinit.BuildCloudConfig()` calls; workloads call this instead of the package-level function - -**`internal/workloads/cpu.go`** -- `CPUWorkload` struct embedding `BaseWorkload` -- Installs `stress-ng`, creates systemd service: `stress-ng --cpu 0 --cpu-method all --timeout 0` -- No extra volumes/disks/services - -**`internal/workloads/memory.go`** -- `MemoryWorkload` struct embedding `BaseWorkload` -- Installs `stress-ng`, creates systemd service: `stress-ng --vm 1 --vm-bytes 80% --vm-method all --timeout 0` -- `--vm 1` (not `--vm 0`): Unlike `--cpu 0` which means "use all CPUs" (a parallelism concern), `--vm 0` would match CPU count and spawn multiple independent allocators. Memory pressure is about capacity, not parallelism, so a single worker (`--vm 1`) that targets a percentage of total memory is the correct approach. -- `--vm-bytes 80%`: Provides meaningful pressure without triggering immediate OOM-kill. At 80%, the guest OS remains functional while the memory subsystem is under sustained load. Higher values (90-100%) risk the OOM killer terminating stress-ng or other guest processes. -- `--vm-method all` rotates through all memory stressor methods (mmap/write/munmap patterns) for broad coverage of memory subsystem behavior -- No extra volumes/disks/services (structurally identical to CPU workload) - -**`internal/workloads/disk.go`** -- `DiskWorkload` struct embedding `BaseWorkload` -- Installs `fio`, writes two profiles (`mixed-rw.fio`: 4K random R/W 70/30 mix; `seq-write.fio`: 128K sequential write) -- Systemd service alternates between profiles with 10s pauses -- DataVolumeTemplate for `/mnt/data` - -### Tests (TDD) - -**`internal/workloads/workload_test.go`** (Ginkgo): -```go -Describe("BaseWorkload", func() { - It("should return nil for ExtraVolumes", ...) - It("should return nil for ExtraDisks", ...) - It("should return nil for DataVolumeTemplates", ...) - It("should return false for RequiresService", ...) - It("should return nil for ServiceSpec", ...) - It("should return 1 for VMCount", ...) -}) -``` - -**`internal/workloads/cpu_test.go`** (Ginkgo): -```go -Describe("CPUWorkload", func() { - It("should return 'cpu' for Name", ...) - It("should include stress-ng in packages", ...) - It("should include systemd service in cloud-init", ...) - It("should produce valid YAML", ...) - It("should have no extra disks", ...) - It("should have no service", ...) - It("should reflect config in VMResources", ...) -}) -``` - -**`internal/workloads/memory_test.go`** (Ginkgo): -```go -Describe("MemoryWorkload", func() { - It("should return 'memory' for Name", ...) - It("should include stress-ng in packages", ...) - It("should include systemd service with --vm flag", ...) - It("should include --vm-bytes 80% in stress-ng args", ...) - It("should include --vm-method all in stress-ng args", ...) - It("should produce valid YAML", ...) - It("should have no extra disks", ...) - It("should have no extra volumes", ...) - It("should have no data volume templates", ...) - It("should not require service", ...) - It("should reflect config in VMResources", ...) -}) -``` - -**`internal/workloads/disk_test.go`** (Ginkgo): -```go -Describe("DiskWorkload", func() { - It("should return 'disk' for Name", ...) - It("should include fio in packages", ...) - It("should include fio profiles in write_files", ...) - It("should have data volume template", ...) - It("should have extra disk for /mnt/data", ...) - It("should not require service", ...) -}) -``` - -### Verification - -- `go test ./internal/workloads/...` all pass -- Cloud-init output is valid YAML -- `BaseWorkload` defaults work via embedding - -### Commits - -- `feat: add Workload interface and BaseWorkload defaults with SSH support` -- `feat: add CPU stress-ng workload` -- `feat: add memory stress-ng VM pressure workload` -- `feat: add disk fio workload` - ---- - -## Phase 7: Database and Network Workloads - -**Goal:** Implement the two more complex workloads. Database needs DataVolume + multi-step cloud-init. Network needs two VMs + K8s Service. - -### Files to Create - -**`internal/workloads/database.go`** -- `DatabaseWorkload` struct embedding `BaseWorkload` -- Installs `postgresql-server`, writes setup script (format data disk, mount, initdb, create pgbench DB with scale 50) -- Uses `ExecStartPre` for one-time database initialization (format, mount, initdb) and `ExecStart` for the continuous benchmark loop — this separation ensures init runs once and the benchmark restarts cleanly on failure -- Systemd service loops `pgbench -c 10 -j 2 -T 300` with 10s pauses -- DataVolumeTemplate for `/var/lib/pgsql/data` - -**`internal/workloads/network.go`** -- `NetworkWorkload` struct embedding `BaseWorkload` + `Namespace string` -- Constructor accepts `namespace` parameter (beyond standard `WorkloadConfig`) for DNS name construction — this is the primary reason the registry uses functional options -- `VMCount() → 2` (server + client) -- Implements `MultiVMWorkload` interface with `UserdataForRole(role, namespace)` -- Server role: installs `iperf3`, runs `iperf3 -s` via systemd -- Client role: installs `iperf3`, loops `iperf3 -c -t 60 -P 4 --bidir` via systemd -- `RequiresService() → true` -- `ServiceSpec(namespace)`: ClusterIP targeting server VM by label `virtwork/role: server`, port 5201 -- Client uses DNS: `virtwork-iperf3-server..svc.cluster.local` - -### Design Note — Network Workload and Multi-VM Detection - -The `Workload` interface defines `CloudInitUserdata()` returning a single string. The network workload needs different userdata for server vs. client. Solution: the orchestration layer checks `VMCount() > 1` and type-asserts to the `MultiVMWorkload` interface to call `UserdataForRole()`. This keeps detection generic — any future multi-VM workload with per-role userdata works without changing orchestration code. - -```go -// MultiVMWorkload extends Workload for workloads that need per-role userdata. -type MultiVMWorkload interface { - Workload - UserdataForRole(role string, namespace string) (string, error) -} -``` - -The orchestration adds a `virtwork/role` label to each VM (e.g., `server`, `client`). This label is what the K8s Service selector uses to route traffic to the server VM. The Service must be created before the client VM so DNS resolves correctly. - -### Design Note — Registry Functional Options - -The `NetworkWorkload` constructor needs a `namespace` parameter beyond the standard `WorkloadConfig`. Rather than adding every possible workload-specific parameter to the registry's `Get()` signature, use functional options: - -```go -type Option func(*RegistryOpts) -func WithNamespace(ns string) Option { return func(o *RegistryOpts) { o.Namespace = ns } } - -func (r Registry) Get(name string, config WorkloadConfig, opts ...Option) (Workload, error) { ... } -``` - -This keeps the registry interface extensible without breaking changes when future workloads need additional constructor parameters. - -### Tests (TDD) - -**`internal/workloads/database_test.go`** (Ginkgo): -```go -Describe("DatabaseWorkload", func() { - It("should return 'database' for Name", ...) - It("should include postgresql-server in packages", ...) - It("should include setup script in cloud-init", ...) - It("should include pgbench systemd service", ...) - It("should have data volume template", ...) - It("should have extra disk for data", ...) - It("should not require service", ...) -}) -``` - -**`internal/workloads/network_test.go`** (Ginkgo): -```go -Describe("NetworkWorkload", func() { - It("should return 'network' for Name", ...) - It("should return 2 for VMCount", ...) - It("should require service", ...) - It("should produce server userdata with iperf3 -s", ...) - It("should produce client userdata with DNS name", ...) - It("should have service spec with correct port", ...) - It("should have service spec with correct selector", ...) -}) -``` - -### Verification - -- `go test ./internal/workloads/...` all pass -- Database cloud-init includes disk mount and PostgreSQL init sequence -- Network workload produces distinct server/client configs -- Service spec uses correct label selector - -### Commits - -- `feat: add database PostgreSQL+pgbench workload` -- `feat: add network iperf3 server/client workload` - ---- - -## Phase 8: Workload Registry - -**Goal:** Complete Layer 3 with a registry for dynamic workload lookup by name. - -### Files to Create - -**`internal/workloads/registry.go`** -- `Registry` map type: `map[string]WorkloadFactory` -- `Option` type and `RegistryOpts` struct for functional options (e.g., `WithNamespace()`, `WithSSHCredentials()`) -- `DefaultRegistry() Registry` — returns registry with all five workloads registered (cpu, memory, database, network, disk) -- `(r Registry) Get(name string, config WorkloadConfig, opts ...Option) (Workload, error)` — returns error listing available names for unknown workloads -- `(r Registry) List() []string` — returns sorted workload names -- `AllWorkloadNames` package-level variable (sorted: cpu, database, disk, memory, network) - -### Tests (TDD) - -**`internal/workloads/registry_test.go`** (Ginkgo): -```go -Describe("Registry", func() { - It("should have 5 entries registered", ...) - It("should return CPU workload by name", ...) - It("should return memory workload by name", ...) - It("should return database workload by name", ...) - It("should return network workload by name", ...) - It("should return disk workload by name", ...) - It("should return error for unknown name with available names", ...) - It("should list all names sorted alphabetically", ...) - It("should create workloads with provided config", ...) - It("should pass namespace option to network workload", ...) -}) -``` - -**Implementation lesson from Python:** When adding a new workload, expect a ripple effect in registry tests (entry count, name lists) and orchestration BDD tests (total VM count assertions). Well-written tests catch this cascade immediately — the fixes are mechanical (updating expected counts and lists). - -### Verification - -- `go test ./internal/workloads/...` all pass -- All workloads discoverable by name string -- Unknown names produce clear error - -### Commits - -- `feat: add workload registry with dynamic lookup` - ---- - -## Phase 9: Cleanup - -**Goal:** Implement label-based resource teardown. Error-tolerant — individual failures don't abort. - -**Key design decision from Python implementation:** The cleanup module implements deletion inline rather than reusing existing `DeleteVM()` and `DeleteManagedServices()` functions. Those functions use fail-fast error semantics (correct for create-time operations), while cleanup needs continue-on-error semantics. Different error handling requirements justify separate implementation — sometimes duplication is the lesser evil. - -### Files to Create - -**`internal/cleanup/cleanup.go`** -- `CleanupResult` struct: `VMsDeleted int`, `ServicesDeleted int`, `NamespaceDeleted bool`, `Errors []error` -- `CleanupAll(ctx context.Context, c client.Client, namespace string, deleteNamespace bool) (*CleanupResult, error)` - - List VMs by label `managed-by=virtwork` via `client.MatchingLabels` - - Delete each VM individually — errors appended to `Errors` slice, not fatal - - Delete managed services with same error-tolerance pattern - - Optionally delete namespace (last step, after resources are drained) - - Returns summary with accurate counts of successful deletions - -### Tests (TDD) - -**`internal/cleanup/cleanup_test.go`** (Ginkgo): -```go -Describe("CleanupAll", func() { - It("should delete VMs by managed-by label", ...) - It("should tolerate individual VM deletion errors", ...) - It("should delete services by managed-by label", ...) - It("should tolerate individual service deletion errors", ...) - It("should not delete namespace by default", ...) - It("should delete namespace when flagged", ...) - It("should tolerate namespace deletion error", ...) - It("should report accurate counts for successful deletions", ...) - It("should handle empty namespace gracefully", ...) - It("should use correct managed-by=virtwork label selector", ...) -}) -``` - -### Verification - -- `go test ./internal/cleanup/...` all pass -- Individual deletion failures don't abort -- Summary has accurate counts - -### Commits - -- `feat: add cleanup package with error-tolerant teardown` - ---- - -## Phase 10: CLI and Entry Point - -**Goal:** Build the Cobra command tree, orchestration flow, and wire up `cmd/virtwork/main.go`. This is the integration point for all packages. - -### Files to Create/Modify - -**`cmd/virtwork/main.go`** -- `rootCmd` with persistent flags: `--namespace`, `--kubeconfig`, `--config`, `--verbose` -- `runCmd` subcommand: `--workloads`, `--vm-count`, `--cpu-cores`, `--memory`, `--disk-size`, `--container-disk-image`, `--no-wait`, `--timeout`, `--dry-run`, `--ssh-user`, `--ssh-password`, `--ssh-key`, `--ssh-key-file` -- `cleanupCmd` subcommand: `--delete-namespace` -- `func main()` calls `rootCmd.Execute()` - -**Run flow (in `runCmd.RunE`) — dry-run early-return pattern:** -1. Load config via Viper -2. Build workload instances from registry, passing SSH credentials and namespace via functional options -3. Build VM specs for all workloads (handles multi-VM workloads via `MultiVMWorkload` type assertion) -4. Check dry-run → if true: print specs as YAML, return -5. Connect to cluster via `cluster.Connect()` -6. `resources.EnsureNamespace()` -7. For each enabled workload: create Service if `RequiresService()` (before VMs for DNS), spawn VM creation goroutines via errgroup -8. `errgroup.Wait()` for all VM creates -9. If not `--no-wait`: `wait.WaitForAllVMsReady()` via errgroup -10. Print summary table - -**Cleanup flow (in `cleanupCmd.RunE`):** -1. Load config via Viper -2. Connect to cluster via `cluster.Connect()` -3. `cleanup.CleanupAll()` — delete all labeled resources with error tolerance -4. Print cleanup summary - -Each subcommand is a linear sequence — no nested if/else chains. - -### Tests (TDD) - -**`cmd/virtwork/main_test.go`** (Ginkgo): - -Argument parsing tests: -```go -Describe("Run command flags", func() { - It("should have default namespace", ...) - It("should accept custom namespace", ...) - It("should accept workloads CSV", ...) - It("should accept vm-count", ...) - It("should accept cpu-cores", ...) - It("should accept memory", ...) - It("should accept disk-size", ...) - It("should accept dry-run flag", ...) - It("should accept no-wait flag", ...) - It("should accept verbose flag", ...) - It("should accept timeout", ...) - It("should accept config file", ...) - It("should accept ssh-user flag", ...) - It("should accept ssh-password flag", ...) - It("should accept ssh-key flag (repeatable)", ...) - It("should accept ssh-key-file flag (repeatable)", ...) -}) -``` - -Orchestration tests (with fake client.Client): -```go -Describe("Run orchestration", func() { - Context("dry-run mode", func() { - It("should skip cluster connection", ...) - It("should print specs to stdout", ...) - }) - Context("normal mode", func() { - It("should create namespace", ...) - It("should create VMs for each workload", ...) - It("should create service for network workload", ...) - It("should wait for readiness", ...) - It("should skip wait when --no-wait", ...) - }) -}) - -Describe("Cleanup command", func() { - It("should delete managed resources", ...) - It("should print summary", ...) -}) -``` - -BDD-style scenarios (Ginkgo Describe/Context/It): -```go -Describe("CLI end-to-end scenarios", func() { - Context("when running with --dry-run --workloads cpu", func() { - It("should not attempt cluster connection", ...) - It("should print VM specs to stdout", ...) - }) - Context("when running with --cleanup", func() { - It("should delete all managed VMs", ...) - It("should print a cleanup summary", ...) - }) - Context("when running with default arguments", func() { - // Default run creates 6 VMs: cpu=1 + memory=1 + disk=1 + database=1 + network=2 - It("should create VMs for all workloads", ...) - It("should begin readiness polling", ...) - }) -}) -``` - -### Verification - -- `go test ./...` all pass (full suite) -- `ginkgo -r` all pass -- `go run ./cmd/virtwork --help` shows all flags -- `go run ./cmd/virtwork run --dry-run` prints YAML without a cluster -- `go build -o virtwork ./cmd/virtwork` produces binary - -### Commits - -- `feat: add Cobra command tree with run and cleanup subcommands` -- `feat: add run orchestration with errgroup concurrency` -- `feat: wire up main.go entry point` -- `test: add BDD scenarios for CLI` - ---- - -## Dependencies - -### Runtime - -| Dependency | Purpose | -|------------|---------| -| `github.com/spf13/cobra` | CLI command tree | -| `github.com/spf13/viper` | Config priority chain (flags > env > file > defaults) | -| `gopkg.in/yaml.v3` | Cloud-init YAML generation, config file parsing | -| `sigs.k8s.io/controller-runtime` | High-level K8s client with scheme-based typed operations | -| `k8s.io/api` | Core K8s API types (Namespace, Service) | -| `k8s.io/apimachinery` | K8s meta types, errors, labels | -| `k8s.io/client-go` | REST config, kubeconfig loading, retry utilities | -| `kubevirt.io/api` | KubeVirt VirtualMachine/VirtualMachineInstance types | -| `kubevirt.io/containerized-data-importer-api` | CDI DataVolume types | -| `golang.org/x/sync` | `errgroup` for structured concurrent operations | -| `github.com/mattn/go-sqlite3` | CGo SQLite3 driver for audit database | -| `github.com/google/uuid` | UUID generation for run ID labels | - -### Development - -| Dependency | Purpose | -|------------|---------| -| `github.com/onsi/ginkgo/v2` | BDD test framework (Describe/Context/It) | -| `github.com/onsi/gomega` | Expressive test matchers | - -### Test Configuration - -Build tags for test levels: - -```go -//go:build integration -// +build integration - -package integration_test -``` - -Run specific test levels: - -```bash -# Unit tests only (default, no build tag) -go test ./... - -# Integration tests (alongside source, requires cluster) -go test -tags integration ./internal/... - -# E2E tests (separate directory, requires cluster + binary) -go test -tags e2e ./tests/e2e/... - -# All tests -go test -tags "integration e2e" ./... - -# Via Ginkgo -ginkgo -r -ginkgo -r --build-tags integration ./internal/ -ginkgo -r --build-tags e2e ./tests/e2e/ -``` - ---- - -## Phase 11: Integration and E2E Tests - -**Goal:** Add integration tests alongside source code with `//go:build integration` tags and E2E acceptance tests in `tests/e2e/` with `//go:build e2e` tags. - -### Test Architecture - -| Category | Location | Build Tag | What it tests | -|----------|----------|-----------|---------------| -| Integration | `internal/*/_integration_test.go` | `integration` | Individual packages against a real KubeVirt cluster | -| E2E | `tests/e2e/*.go` | `e2e` | CLI binary as a black box (deploy, cleanup, dry-run) | -| Helpers | `internal/testutil/` | (none) | Shared utilities: namespace generation, cluster connect, binary execution | - -### Files Created - -**Shared helpers:** -- `internal/testutil/testutil.go` — `UniqueNamespace()`, `MustConnect()`, `CleanupNamespace()`, `ManagedLabels()`, `DefaultVMOpts()`, `EnsureTestNamespace()`, `WaitForVMRunning()` -- `internal/testutil/binary.go` — `BinaryPath()`, `RunVirtwork()` for E2E binary execution - -**Integration tests (5 files):** -- `internal/cluster/cluster_integration_test.go` — Real cluster connectivity and scheme validation -- `internal/resources/resources_integration_test.go` — Real namespace/service/secret CRUD and idempotency -- `internal/vm/vm_integration_test.go` — Real VirtualMachine create/delete/list -- `internal/wait/wait_integration_test.go` — Real VMI readiness polling (Label("slow")) -- `internal/cleanup/cleanup_integration_test.go` — Real label-based cleanup with error tolerance - -**E2E tests (5 files):** -- `tests/e2e/e2e_suite_test.go` — Ginkgo bootstrap, binary build in `BeforeSuite` -- `tests/e2e/dryrun_test.go` — `virtwork run --dry-run` scenarios (no cluster needed) -- `tests/e2e/run_test.go` — `virtwork run` with real cluster deployment -- `tests/e2e/cleanup_test.go` — `virtwork cleanup` after deployment -- `tests/e2e/fullcycle_test.go` — Deploy → verify → cleanup cycles - -### Key Design Decisions - -- Integration tests share existing `*_suite_test.go` runners (no build tag on suite files) -- Each test creates a unique namespace via `UniqueNamespace()` with `DeferCleanup` teardown -- Slow tests (VM boot ~60-120s) use Ginkgo `Label("slow")` for filtering -- E2E tests invoke the binary via `os/exec`, verifying stdout/stderr/exit code -- `internal/testutil/` is importable by both integration and E2E tests (same Go module) - -### Verification - -```bash -# Unit tests unchanged -go test ./... - -# Integration tests (requires cluster) -go test -tags integration ./internal/... - -# E2E tests (requires cluster + binary) -go test -tags e2e ./tests/e2e/... - -# All tests -go test -tags "integration e2e" ./... -``` - -### Commits - -- `test: add shared test helpers for integration and E2E tests` -- `test: add integration tests for cluster, resources, vm, wait, cleanup` -- `test: add E2E test suite with dry-run, run, cleanup, and full-cycle scenarios` -- `docs: add integration and E2E test documentation` - ---- - -## Phase 12: Audit System - -**Goal:** Add SQLite-based audit tracking for all executions. Every `run` and `cleanup` gets a database record with timestamps, configuration, and outcome details. - -### Files Created - -**`internal/audit/schema.go`** -- DDL for 5 tables: `audit_log`, `workload_details`, `vm_details`, `resource_details`, `events` -- Indexes on foreign keys and common query columns -- Foreign key constraints between all detail tables and `audit_log` - -**`internal/audit/records.go`** -- `WorkloadRecord`, `VMRecord`, `ResourceRecord`, `EventRecord` structs -- Data transfer objects for audit operations - -**`internal/audit/audit.go`** -- `Auditor` interface with 13 methods covering full execution lifecycle -- `SQLiteAuditor` implementation with WAL mode and foreign key enforcement -- `NoOpAuditor` for when audit is disabled -- `NewAuditor(dbPath)` constructor, `NewNoOpAuditor()` constructor - -**`internal/audit/audit_suite_test.go`** — Ginkgo bootstrap -**`internal/audit/audit_test.go`** — 14 tests covering: -- Schema creation (tables + indexes) -- Full execution lifecycle (start → workloads → VMs → events → complete) -- Failure with error summary -- Cleanup linking (single and multi-run) -- Cleanup counts recording -- VM and resource deletion tracking -- Events with FK references -- Concurrent writes (20 goroutines) -- SSH auth boolean tracking -- NoOpAuditor does nothing without error - -### Files Modified - -**`internal/constants/constants.go`** -- Added `LabelRunID = "virtwork/run-id"` for run-to-cleanup linking -- Added `DefaultAuditDBPath = "virtwork.db"` - -**`internal/config/config.go`** -- Added `AuditEnabled` (bool, default: true) and `AuditDBPath` (string, default: `virtwork.db`) -- Viper defaults and LoadConfig wiring - -**`internal/cleanup/cleanup.go`** -- Added `runID string` parameter to `CleanupAll` for targeted cleanup -- Added `RunIDs []string` field to `CleanupResult` -- Added `collectRunID()` helper to extract `virtwork/run-id` from resource labels -- When `runID` is provided, adds it to label selector; otherwise collects all unique run IDs - -**`cmd/virtwork/main.go`** -- Added `--audit`/`--no-audit`/`--audit-db` persistent flags -- Added `--run-id` flag on cleanup command -- Run flow: generates UUID, applies `virtwork/run-id` label, records audit events at each step -- Cleanup flow: uses `--run-id` for targeted deletion, links collected run IDs to audit record - -**`.gitignore`** — Added `virtwork.db` - -### Key Design Decisions - -- **`Auditor` interface + `NoOpAuditor`:** Avoids nil checks throughout codebase when audit is disabled -- **WAL journal mode:** Allows concurrent reads during writes -- **Shared cache for `:memory:` tests:** `file::memory:?mode=memory&cache=shared` ensures all connections in the pool see the same database -- **JSON array for `linked_run_ids`:** Supports cleanup-all (multiple runs) while remaining PostgreSQL JSONB compatible -- **No SSH credentials stored:** Security by design — only a boolean `ssh_auth_configured` - -### Verification - -- `go test ./internal/audit/...` — 14 tests pass -- `go test ./...` — full suite passes (no regressions) -- `go vet ./...` — clean - -### Commits - -- `feat: add run-id label and audit DB path constants` -- `feat: add audit config fields (AuditEnabled, AuditDBPath)` -- `feat: add SQLite audit system with 5-table schema` -- `feat: add run-id label and targeted cleanup support` -- `feat: integrate audit tracking into CLI orchestration` -- `chore: add virtwork.db to .gitignore` -- `test: verify all tests pass after audit integration` - ---- - -## Lessons from Python Implementation - -The following patterns and lessons were discovered during the Python (`dynavirt`) implementation and should be applied to the Go build: - -### Testing Patterns - -1. **YAML assertion via parsing** — Never assert on raw YAML strings. Always `yaml.Unmarshal` the output and assert on the parsed structure. YAML serializers may reorder keys, fold long lines, or vary whitespace between implementations. - -2. **Ripple effect on new workloads** — Adding a new workload triggers test failures in: registry tests (entry count, name list), orchestration BDD tests (total VM count assertions). These are mechanical fixes but must be expected. - -3. **Systemd unit separation** — For workloads with both initialization and runtime phases (like database), use `ExecStartPre` for setup and `ExecStart` for the main loop. This is cleaner than a single script with conditional init logic. - -4. **Fio profile separation** — Write fio job profiles as separate `write_files` entries rather than embedding them in the systemd unit. Keeps the systemd service definition readable and the profiles independently verifiable. - -### Architectural Patterns - -5. **Separate subcommands for distinct operations** — `run` and `cleanup` are separate Cobra subcommands, each with its own `RunE` function and flag set. Within `run`, dry-run is an early-return check. Cleanup has fundamentally different error semantics (continue-on-error) and flags (`--delete-namespace`), justifying a separate command rather than a mode flag. - -6. **Error-tolerant vs. fail-fast** — Create-time operations should fail fast (return error immediately). Cleanup operations should accumulate errors and continue. These are fundamentally different error semantics — don't try to unify them by adding flags to shared functions. - -7. **Cross-cutting concern injection via base struct** — When a concern (SSH credentials) must appear in every workload's output, add a helper method to `BaseWorkload` that wraps the underlying pure function. Workloads change one call site (the method name) and the base struct handles the wiring. - -8. **Service-before-VM ordering** — For multi-VM workloads with DNS dependencies, the K8s Service must be created before the client VM. The client's systemd unit uses `Restart=always` to retry until DNS resolves, but the Service must exist for DNS to work at all. - ---- - -## Risk Mitigations - -| Risk | Mitigation | -|------|------------| -| KubeVirt API type versioning | Pin `kubevirt.io/api` version in go.mod; API coordinates isolated in constants package | -| controller-runtime fake client limitations | Use `fake.NewClientBuilder().WithScheme(scheme).WithObjects(...).Build()` for unit tests | -| Cloud-init YAML correctness | Validate all output against `yaml.Unmarshal` in tests | -| Network workload DNS timing | systemd `Restart=always` on client handles server not yet ready | -| Large dependency tree (client-go) | Go modules handle this; build caching keeps iteration fast | -| Race conditions in concurrent tests | Use `ginkgo -race` and `go test -race` in CI | -| HAProxy cold-pool TLS failures | Load balancers may close idle connections silently. controller-runtime handles TLS renegotiation, but be aware of transient 503s on first API call after idle period. Log and retry. | -| Default run creates too many VMs as workloads grow | Each new workload adds VMs to the default set (currently 6: cpu=1 + memory=1 + disk=1 + database=1 + network=2). Consider a curated default subset if workload count grows beyond 6-7. | diff --git a/docs/openshift-virtualization-workload-automation.md b/docs/openshift-virtualization-workload-automation.md deleted file mode 100644 index 3e757b3..0000000 --- a/docs/openshift-virtualization-workload-automation.md +++ /dev/null @@ -1,277 +0,0 @@ -# Plan: OpenShift Virtualization Workload Automation ("virtwork") - -> **Historical snapshot — original design rationale.** -> -> This document captured the motivation and design choices at project inception, when no application code existed yet. It is preserved as context for *why* the layered architecture, the deploy-and-exit model, and the systemd-inside-VM workload lifecycle were chosen. **Implementation has since evolved.** For the current state, see: -> -> - Architecture and design decisions → [architecture.md](architecture.md) -> - Built-in workloads (now nine total) → [README.md](../README.md#workloads), [architecture.md](architecture.md), [chaos-workloads.md](chaos-workloads.md) -> - Developer guide and patterns → [development.md](development.md) -> - Configuration reference → [configuration.md](configuration.md) -> - OpenShift deployment → [deployment.md](deployment.md) - -## Context - -This codebase (`virtwork`) is a Go CLI tool for creating VMs on an OpenShift cluster with OpenShift Virtualization (CNV) already installed. The goal is to run continuous, varied workloads (CPU, memory, database, network, disk I/O) inside those VMs so that monitoring tools (Prometheus, Grafana, etc.) have realistic metrics to observe. The codebase currently has a `go.mod` and documentation — no application code yet. - ---- - -## Project Structure - -``` -virtwork/ -├── cmd/ -│ └── virtwork/ -│ └── main.go # Entry point: Cobra root command -├── internal/ -│ ├── config/ -│ │ └── config.go # Config struct, Viper priority chain -│ ├── constants/ -│ │ └── constants.go # API groups, versions, label keys, defaults -│ ├── cluster/ -│ │ └── cluster.go # controller-runtime client init -│ ├── cloudinit/ -│ │ └── cloudinit.go # Cloud-init YAML builder -│ ├── vm/ -│ │ └── vm.go # VM spec construction + CRUD -│ ├── resources/ -│ │ └── resources.go # Namespace + Service helpers -│ ├── wait/ -│ │ └── wait.go # Poll VMI status until Running -│ ├── cleanup/ -│ │ └── cleanup.go # Teardown by label selector -│ └── workloads/ -│ ├── workload.go # Workload interface -│ ├── registry.go # Registry map + lookup -│ ├── cpu.go # stress-ng CPU continuous workload -│ ├── memory.go # stress-ng VM memory pressure workload -│ ├── database.go # PostgreSQL + pgbench loop -│ ├── network.go # iperf3 server/client pair -│ └── disk.go # fio mixed I/O profiles -├── tests/ -│ ├── integration/ # Integration tests (build tag) -│ └── e2e/ # E2E tests (build tag) -├── docs/ -├── go.mod -├── go.sum -└── CLAUDE.md -``` - ---- - -## Implementation Steps - -### 1. Update `go.mod` and add dependencies - -``` -module virtwork - -go 1.25 - -require ( - github.com/spf13/cobra v1.9+ - github.com/spf13/viper v1.20+ - gopkg.in/yaml.v3 v3.0+ - sigs.k8s.io/controller-runtime v0.20+ - kubevirt.io/api v1.5+ - kubevirt.io/containerized-data-importer-api v1.62+ - k8s.io/api v0.32+ - k8s.io/apimachinery v0.32+ - k8s.io/client-go v0.32+ -) -``` - -Development dependencies (test tooling): -``` -github.com/onsi/ginkgo/v2 v2.23+ -github.com/onsi/gomega v1.37+ -``` - -### 2. Create `internal/constants/constants.go` - -- KubeVirt API coordinates: group `kubevirt.io`, version `v1`, plurals for `virtualmachines` and `virtualmachineinstances` -- CDI API coordinates: group `cdi.kubevirt.io`, version `v1beta1`, plural for `datavolumes` -- Default container disk image: `quay.io/containerdisks/fedora:41` -- Standard Kubernetes labels (`app.kubernetes.io/name`, `managed-by`, `component`) -- Default values: namespace=`virtwork`, cpu=2, memory=`2Gi`, disk=`10Gi` -- Polling defaults: timeout=600s, interval=15s - -### 3. Create `internal/config/config.go` - -- `WorkloadConfig` struct: `Enabled bool`, `VMCount int`, `CPUCores int`, `Memory string` -- `Config` struct: `Namespace`, `ContainerDiskImage`, `DataDiskSize`, workloads map, `KubeconfigPath`, `CleanupMode`, `WaitForReady`, `ReadyTimeoutSeconds`, `DryRun`, `Verbose`, `SSHUser`, `SSHPassword`, `SSHAuthorizedKeys` -- Viper-based loading: Cobra flags > env vars (`VIRTWORK_*`) > YAML config file > defaults -- Scalar fields map directly through Viper; list fields (`SSHAuthorizedKeys`) need special handling at each config layer (YAML=list, env=comma-separated, CLI=merged from `--ssh-key` and `--ssh-key-file`) - -### 4. Create `internal/cluster/cluster.go` - -- `Connect(kubeconfigPath string) (client.Client, error)` function -- Uses controller-runtime's `client.New()` with rest config -- Try in-cluster config first via `rest.InClusterConfig()`, fall back to `clientcmd.BuildConfigFromFlags()` -- Registers KubeVirt and CDI types in the runtime scheme - -### 5. Create `internal/cloudinit/cloudinit.go` - -- `BuildCloudConfig(opts CloudConfigOpts) (string, error)` -- `CloudConfigOpts` struct: `Packages []string`, `WriteFiles []WriteFile`, `RunCmd [][]string`, `Extra map[string]interface{}`, `SSHUser string`, `SSHPassword string`, `SSHAuthorizedKeys []string` -- Returns `#cloud-config\n` + YAML marshal -- Omits keys with empty/nil values -- When `SSHUser` is set, emits a `users` block with sudo access, shell, and optional password/authorized keys -- `lock_passwd: true` when no password (key-only auth); `lock_passwd: false` + `plain_text_passwd` when password provided -- `ssh_pwauth: true` at top level only when password is set - -### 6. Create workloads - -**`internal/workloads/workload.go`** — Interfaces and base struct: -```go -type Workload interface { - Name() string - CloudInitUserdata() (string, error) - VMResources() VMResourceSpec - ExtraVolumes() []kubevirtv1.Volume - ExtraDisks() []kubevirtv1.Disk - DataVolumeTemplates() []cdiv1beta1.DataVolumeTemplateSpec - RequiresService() bool - ServiceSpec(namespace string) *corev1.Service - VMCount() int -} - -// MultiVMWorkload extends Workload for workloads needing per-role userdata (e.g., network). -type MultiVMWorkload interface { - Workload - UserdataForRole(role string, namespace string) (string, error) -} -``` - -**SSH credential passthrough:** `BaseWorkload` stores optional SSH fields (`SSHUser`, `SSHPassword`, `SSHAuthorizedKeys`) and provides a `BuildCloudConfig()` helper method that injects them into every `cloudinit.BuildCloudConfig()` call. Concrete workloads call `w.BuildCloudConfig(opts)` instead of `cloudinit.BuildCloudConfig(opts)` directly — a single point of injection for the cross-cutting SSH concern. The registry passes SSH credentials via functional options on the constructor. - -**`internal/workloads/cpu.go`** — stress-ng CPU -- Installs `stress-ng`, creates systemd service -- `stress-ng --cpu 0 --cpu-method all --timeout 0` (all CPUs, varied methods, runs forever) - -**`internal/workloads/memory.go`** — stress-ng VM memory pressure -- Installs `stress-ng`, creates systemd service -- `stress-ng --vm 1 --vm-bytes 80% --vm-method all --timeout 0` (single VM worker, 80% of available memory, rotates all memory stressor methods) -- 80% target provides meaningful pressure without triggering immediate OOM-kill -- No extra volumes, disks, or services needed - -**`internal/workloads/database.go`** — PostgreSQL + pgbench -- Needs a blank DataVolume for `/var/lib/pgsql/data` -- Cloud-init: install `postgresql-server`, setup script formats/mounts data disk, inits DB, creates `pgbench` database with scale factor 50 -- Uses `ExecStartPre` for database initialization (format, mount, initdb) and `ExecStart` for the benchmark loop — separating one-time setup from the continuous workload -- Systemd service loops: `pgbench -c 10 -j 2 -T 300` (5-min bursts, 10s pause) - -**`internal/workloads/network.go`** — iperf3 server/client -- Always creates 2 VMs (server + client) -- Creates a Kubernetes `Service` selecting the server VM by label `virtwork/role: server` -- Server: `iperf3 -s` via systemd (runs forever) -- Client: systemd loops `iperf3 -c -t 60 -P 4 --bidir` (60s tests, 15s pause) -- Client uses DNS name `virtwork-iperf3-server..svc.cluster.local` — no IP polling needed -- Retry-on-fail: systemd `Restart=always` handles server not yet ready -- Constructor accepts `namespace` parameter (in addition to `WorkloadConfig`) for DNS name construction - -**`internal/workloads/disk.go`** — fio -- Needs a blank DataVolume for `/mnt/data` -- Two fio profiles written via cloud-init as separate `write_files` entries (bench script separated from systemd unit for clarity): - - `mixed-rw.fio`: 4K random R/W, 70/30 mix, 4 jobs, 5 min - - `seq-write.fio`: 128K sequential write, 2 jobs, 5 min -- Systemd service alternates between profiles with 10s pauses - -### 7. Create `internal/vm/vm.go` - -- `BuildVMSpec(opts VMSpecOpts) *kubevirtv1.VirtualMachine` - - Uses `containerDisk` for OS (fast pull, ephemeral root is fine for workload VMs) - - `cloudInitNoCloud` with embedded userdata - - Masquerade networking (`pod` network) - - Virtio bus for all disks - - `spec.Running: true` to auto-start -- `BuildDataVolumeTemplate(name, size string) cdiv1beta1.DataVolumeTemplateSpec` for blank data disks -- `CreateVM(ctx context.Context, c client.Client, vm *kubevirtv1.VirtualMachine) error` — AlreadyExists=skip, retry on transient errors -- `DeleteVM(ctx context.Context, c client.Client, name, namespace string) error` -- `ListVMs(ctx context.Context, c client.Client, namespace string, labels map[string]string) ([]kubevirtv1.VirtualMachine, error)` -- `GetVMIPhase(ctx context.Context, c client.Client, name, namespace string) (kubevirtv1.VirtualMachineInstancePhase, error)` -- Retry logic: AlreadyExists=skip, rate-limited/5xx=retry with backoff, NotFound=fatal "CNV not installed?", Unauthorized/Forbidden=fatal auth error - -### 8. Create `internal/resources/resources.go` - -- `EnsureNamespace(ctx context.Context, c client.Client, name string, labels map[string]string) error` — create if not exists (AlreadyExists=skip) -- `CreateService(ctx context.Context, c client.Client, svc *corev1.Service) error` — AlreadyExists=skip -- `DeleteManagedServices(ctx context.Context, c client.Client, namespace string, labels map[string]string) (int, error)` — returns count - -### 9. Create `internal/wait/wait.go` - -- `WaitForVMReady(ctx context.Context, c client.Client, name, namespace string, timeout, interval time.Duration) error` -- Polls VMI phase, returns nil when `phase == Running`, returns error on timeout -- `WaitForAllVMsReady(ctx context.Context, c client.Client, names []string, namespace string, timeout, interval time.Duration) map[string]error` -- Uses `errgroup.Group` for concurrent polling - -### 10. Create `internal/cleanup/cleanup.go` - -- `CleanupAll(ctx context.Context, c client.Client, namespace string, deleteNamespace bool) (*CleanupResult, error)` -- `CleanupResult` struct: `VMsDeleted int`, `ServicesDeleted int`, `NamespaceDeleted bool`, `Errors []error` -- List all VMs with label `app.kubernetes.io/managed-by=virtwork` -- Delete each VM (errors logged, not fatal) -- Delete managed Services -- Optionally delete namespace - -### 11. Create `cmd/virtwork/main.go` with Cobra commands - -- Root command with persistent flags: `--namespace`, `--kubeconfig`, `--config`, `--verbose` -- `run` subcommand (default): `--workloads`, `--vm-count`, `--cpu-cores`, `--memory`, `--disk-size`, `--container-disk-image`, `--no-wait`, `--timeout`, `--dry-run`, `--ssh-user`, `--ssh-password`, `--ssh-key`, `--ssh-key-file` -- `cleanup` subcommand: `--delete-namespace` -- `run` orchestration flow (`runCmd.RunE`): - 1. Load config via Viper (Cobra flags > env > file > defaults) - 2. If dry-run: generate specs, print as YAML, exit - 3. Connect to cluster via controller-runtime - 4. Ensure namespace - 5. For each enabled workload: - - Get workload from registry, passing SSH credentials via functional options - - If `VMCount() > 1` and workload implements `MultiVMWorkload` interface: iterate roles (server, client), call `UserdataForRole()` for each, add `virtwork/role` label per VM - - Otherwise: single `CloudInitUserdata()` call - - Create Service if `RequiresService()` (must exist before client VM for DNS resolution) - - Build VM spec(s), create VM(s) — parallel via errgroup - 6. If not `--no-wait`: poll for readiness via errgroup - 7. Print summary table -- `cleanup` orchestration flow (`cleanupCmd.RunE`): - 1. Load config via Viper - 2. Connect to cluster - 3. Delete all labeled resources (`cleanup.CleanupAll()`) - 4. Print cleanup summary - ---- - -## Key Design Decisions - -| Decision | Choice | Rationale | -|----------|--------|-----------| -| Boot disk | `containerDisk` | Fast (kubelet image pull), cached on nodes, no CDI import wait. Ephemeral root is fine for workload VMs. | -| Data disk | Blank `DataVolume` | Formatted on first boot by cloud-init. Only needed for database and fio workloads. | -| Workload lifecycle | systemd services | Survive reboots, auto-restart on failure, proper logging via journald. | -| Network coordination | K8s Service + DNS | No IP polling from Go. Client retries via systemd `Restart=always` until server is ready. | -| Cleanup tracking | Label selectors | No state file needed. Works even if tool crashed mid-creation. | -| Auth | In-cluster first, kubeconfig fallback | Works both inside pods (CI/CD) and from developer machines. | -| Concurrency | goroutines + errgroup | Native Go concurrency. Parallel VM creation and readiness polling with structured error handling. | -| K8s client | controller-runtime `client.Client` | Higher-level than raw client-go. Typed operations with KubeVirt/CDI API types. Common in OpenShift ecosystem. | -| CLI framework | Cobra + Viper | Standard Go CLI stack. Viper integrates with Cobra for config priority chain. | -| Testing | Ginkgo v2 + Gomega | BDD framework with expressive matchers. Native Describe/Context/It blocks. | -| Idempotency | AlreadyExists = skip | Safe to re-run. Enables declarative approach. | -| Retry | Backoff for rate-limited/5xx | Handles transient cluster issues. NotFound/Unauthorized/Forbidden are fatal. | -| Multi-VM detection | `VMCount() > 1` + `MultiVMWorkload` type assertion | Keeps orchestration generic; future multi-VM workloads work without changing orchestration code. | -| SSH credential injection | `BaseWorkload.BuildCloudConfig()` helper | Each workload calls one helper method instead of passing SSH args individually. Single-word change per workload. | -| Cleanup error semantics | Inline iteration with per-resource error capture | Existing delete functions raise on error (correct for create-time). Cleanup needs continue-on-error — different semantics justify separate implementation. | -| Subcommand separation | `run` and `cleanup` are separate Cobra subcommands; dry-run is an early-return within `run` | Distinct flag sets and error semantics (fail-fast vs continue-on-error) justify separate commands. Each is a linear sequence. | -| Registry functional options | `Get(name, config, ...Option)` | NetworkWorkload needs `namespace` beyond standard `WorkloadConfig`. Functional options keep the registry interface extensible without breaking changes. | - ---- - -## Verification - -1. **Dry run**: `go run ./cmd/virtwork run --dry-run` — prints all VM specs as YAML, no cluster needed -2. **Create resources**: `go run ./cmd/virtwork run --namespace virtwork-test --verbose` -3. **Check VMs**: `oc get vm -n virtwork-test` — all VMs should show `Running` -4. **Check VMIs**: `oc get vmi -n virtwork-test` — all VMIs should show `Running` with IPs -5. **Verify workloads inside VMs**: `virtctl console virtwork-cpu-0` then `systemctl status virtwork-cpu` -6. **Verify SSH access**: `virtctl ssh --ssh-key ~/.ssh/id_rsa virtwork@virtwork-cpu-0` (when `--ssh-key-file` or `--ssh-password` was provided) -7. **Verify monitoring**: Check Prometheus/Grafana for CPU, memory, disk, network, and database metrics from the VMs -8. **Cleanup**: `go run ./cmd/virtwork cleanup --namespace virtwork-test` — all resources removed -9. **Idempotency**: Run create twice — second run should skip existing resources (AlreadyExists handled) From 83a0d8923ac65146c8db905ac3c04693ef0c786a Mon Sep 17 00:00:00 2001 From: Melvin Hillsman Date: Wed, 10 Jun 2026 21:29:40 -0500 Subject: [PATCH 2/4] style: apply gofumpt and golines formatting Signed-off-by: Melvin Hillsman --- internal/orchestrator/orchestrator.go | 62 +++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/internal/orchestrator/orchestrator.go b/internal/orchestrator/orchestrator.go index 6157019..db11b7d 100644 --- a/internal/orchestrator/orchestrator.go +++ b/internal/orchestrator/orchestrator.go @@ -86,9 +86,17 @@ func (ro *RunOrchestrator) Run( if err := ro.auditor.RecordEvent(ctx, execID, audit.EventRecord{ EventType: "execution_started", - Message: fmt.Sprintf("Planned %d VMs across %d workloads", len(plans), len(workloadNames)), + Message: fmt.Sprintf( + "Planned %d VMs across %d workloads", + len(plans), + len(workloadNames), + ), }); err != nil { - ro.logger.Warn("audit record failed", slog.String("op", "RecordEvent"), slog.String("error", err.Error())) + ro.logger.Warn( + "audit record failed", + slog.String("op", "RecordEvent"), + slog.String("error", err.Error()), + ) } if cfg.DryRun { @@ -170,7 +178,10 @@ func (ro *RunOrchestrator) planVMs( if fileCfg.Enabled != nil && !*fileCfg.Enabled { if err := ro.auditor.RecordEvent(ctx, execID, audit.EventRecord{ EventType: "workload_skipped", - Message: fmt.Sprintf("Workload %q disabled via config (enabled: false)", name), + Message: fmt.Sprintf( + "Workload %q disabled via config (enabled: false)", + name, + ), }); err != nil { ro.logger.Warn( "audit record failed", @@ -217,7 +228,11 @@ func (ro *RunOrchestrator) planVMs( dvTemplatesForAudit, err := w.DataVolumeTemplates() if err != nil { - return nil, nil, nil, nil, fmt.Errorf("building data volume templates for %q: %w", name, err) + return nil, nil, nil, nil, fmt.Errorf( + "building data volume templates for %q: %w", + name, + err, + ) } wlID, auditErr := ro.auditor.RecordWorkload(ctx, execID, audit.WorkloadRecord{ WorkloadType: name, @@ -605,7 +620,11 @@ func (ro *RunOrchestrator) waitForReadiness( if failures > 0 { for comp, wlID := range auditWorkloadIDs { if failedWorkloads[comp] { - if auditErr := ro.auditor.UpdateWorkloadStatus(ctx, wlID, "failed"); auditErr != nil { + if auditErr := ro.auditor.UpdateWorkloadStatus( + ctx, + wlID, + "failed", + ); auditErr != nil { ro.logger.Warn( "audit record failed", slog.String("op", "UpdateWorkloadStatus"), @@ -613,7 +632,11 @@ func (ro *RunOrchestrator) waitForReadiness( ) } } else { - if auditErr := ro.auditor.UpdateWorkloadStatus(ctx, wlID, "created"); auditErr != nil { + if auditErr := ro.auditor.UpdateWorkloadStatus( + ctx, + wlID, + "created", + ); auditErr != nil { ro.logger.Warn( "audit record failed", slog.String("op", "UpdateWorkloadStatus"), @@ -654,7 +677,14 @@ func (ro *RunOrchestrator) printDryRun( if err != nil { return 0, 0, fmt.Errorf("marshaling Service for %q: %w", name, err) } - _, _ = fmt.Fprintf(ro.writer, "# Service: %s (workload: %s)\n%s\n%s\n", svc.Name, name, string(data), "---") + _, _ = fmt.Fprintf( + ro.writer, + "# Service: %s (workload: %s)\n%s\n%s\n", + svc.Name, + name, + string(data), + "---", + ) svcCount++ } } @@ -687,7 +717,14 @@ func (ro *RunOrchestrator) printDryRun( if err != nil { return 0, 0, fmt.Errorf("marshaling Secret for %q: %w", plans[i].VMName, err) } - _, _ = fmt.Fprintf(ro.writer, "# Secret: %s (workload: %s)\n%s\n%s\n", secretName, plans[i].Component, string(data), "---") + _, _ = fmt.Fprintf( + ro.writer, + "# Secret: %s (workload: %s)\n%s\n%s\n", + secretName, + plans[i].Component, + string(data), + "---", + ) } for _, p := range plans { @@ -711,7 +748,14 @@ func (ro *RunOrchestrator) printDryRun( if err != nil { return 0, 0, fmt.Errorf("marshaling VM spec for %q: %w", p.VMName, err) } - _, _ = fmt.Fprintf(ro.writer, "# VM: %s (workload: %s)\n%s\n%s\n", p.VMName, p.Component, string(data), "---") + _, _ = fmt.Fprintf( + ro.writer, + "# VM: %s (workload: %s)\n%s\n%s\n", + p.VMName, + p.Component, + string(data), + "---", + ) } return svcCount, len(plans), nil } From 57beed1c4e1e193b2147f983614d4cf2962d1cec Mon Sep 17 00:00:00 2001 From: Melvin Hillsman Date: Wed, 10 Jun 2026 21:30:34 -0500 Subject: [PATCH 3/4] feat: add name-based audit deletion tracking to cleanup Signed-off-by: Melvin Hillsman --- internal/audit/audit.go | 218 +++++++++++++++++++++++--- internal/audit/audit_test.go | 146 +++++++++++++++++ internal/cleanup/cleanup.go | 11 ++ internal/cleanup/cleanup_test.go | 43 +++++ internal/orchestrator/cleanup.go | 69 +++++++- internal/orchestrator/cleanup_test.go | 82 ++++++++++ 6 files changed, 546 insertions(+), 23 deletions(-) diff --git a/internal/audit/audit.go b/internal/audit/audit.go index f39136d..b94f566 100644 --- a/internal/audit/audit.go +++ b/internal/audit/audit.go @@ -22,7 +22,11 @@ import ( // Auditor defines the contract for recording execution audit data. type Auditor interface { // StartExecution creates an audit_log row and returns its ID and generated run UUID. - StartExecution(ctx context.Context, cmd string, cfg *config.Config) (executionID int64, runID string, err error) + StartExecution( + ctx context.Context, + cmd string, + cfg *config.Config, + ) (executionID int64, runID string, err error) // CompleteExecution finalises the audit_log row with status and optional error summary. CompleteExecution(ctx context.Context, id int64, status string, errSummary string) error // LinkCleanupToRuns sets linked_run_ids on a cleanup audit_log row. @@ -36,22 +40,53 @@ type Auditor interface { ) error // RecordWorkload inserts a workload_details row. - RecordWorkload(ctx context.Context, executionID int64, w WorkloadRecord) (workloadID int64, err error) + RecordWorkload( + ctx context.Context, + executionID int64, + w WorkloadRecord, + ) (workloadID int64, err error) // UpdateWorkloadStatus updates the status and completed_at of a workload_details row. UpdateWorkloadStatus(ctx context.Context, id int64, status string) error // RecordVM inserts a vm_details row. - RecordVM(ctx context.Context, executionID int64, workloadID int64, v VMRecord) (vmID int64, err error) + RecordVM( + ctx context.Context, + executionID int64, + workloadID int64, + v VMRecord, + ) (vmID int64, err error) // UpdateVMStatus updates phase and status on a vm_details row. UpdateVMStatus(ctx context.Context, id int64, phase string, status string) error // RecordVMDeletion sets deleted_at and status='deleted' on a vm_details row. RecordVMDeletion(ctx context.Context, id int64) error // RecordResource inserts a resource_details row. - RecordResource(ctx context.Context, executionID int64, r ResourceRecord) (resourceID int64, err error) + RecordResource( + ctx context.Context, + executionID int64, + r ResourceRecord, + ) (resourceID int64, err error) // RecordResourceDeletion sets deleted_at and status='deleted' on a resource_details row. RecordResourceDeletion(ctx context.Context, id int64) error + // MarkVMsDeletedByName marks vm_details rows as deleted by VM name and namespace, + // scoped to audit records from the given run IDs. + MarkVMsDeletedByName( + ctx context.Context, + vmNames []string, + namespace string, + runIDs []string, + ) error + // MarkResourcesDeletedByName marks resource_details rows as deleted by type, name, + // and namespace, scoped to audit records from the given run IDs. + MarkResourcesDeletedByName( + ctx context.Context, + resourceType string, + names []string, + namespace string, + runIDs []string, + ) error + // RecordEvent inserts an events row. RecordEvent(ctx context.Context, executionID int64, e EventRecord) error @@ -95,7 +130,11 @@ func now() string { return time.Now().UTC().Format(time.RFC3339) } -func (a *SQLiteAuditor) StartExecution(ctx context.Context, cmd string, cfg *config.Config) (int64, string, error) { +func (a *SQLiteAuditor) StartExecution( + ctx context.Context, + cmd string, + cfg *config.Config, +) (int64, string, error) { runID := uuid.New().String() workloadNames := make([]string, 0, len(cfg.Workloads)) @@ -106,7 +145,8 @@ func (a *SQLiteAuditor) StartExecution(ctx context.Context, cmd string, cfg *con sshConfigured := cfg.SSHPassword != "" || len(cfg.SSHAuthorizedKeys) > 0 - res, err := a.db.ExecContext(ctx, ` + res, err := a.db.ExecContext( + ctx, ` INSERT INTO audit_log ( run_id, command, status, kubeconfig_path, cluster_context, namespace, container_disk_image, default_cpu_cores, default_memory, data_disk_size, @@ -129,7 +169,12 @@ func (a *SQLiteAuditor) StartExecution(ctx context.Context, cmd string, cfg *con return id, runID, nil } -func (a *SQLiteAuditor) CompleteExecution(ctx context.Context, id int64, status string, errSummary string) error { +func (a *SQLiteAuditor) CompleteExecution( + ctx context.Context, + id int64, + status string, + errSummary string, +) error { var errPtr *string if errSummary != "" { errPtr = &errSummary @@ -140,7 +185,11 @@ func (a *SQLiteAuditor) CompleteExecution(ctx context.Context, id int64, status return err } -func (a *SQLiteAuditor) LinkCleanupToRuns(ctx context.Context, cleanupID int64, runIDs []string) error { +func (a *SQLiteAuditor) LinkCleanupToRuns( + ctx context.Context, + cleanupID int64, + runIDs []string, +) error { data, err := json.Marshal(runIDs) if err != nil { return fmt.Errorf("marshaling run IDs: %w", err) @@ -171,8 +220,13 @@ func (a *SQLiteAuditor) RecordCleanupCounts( return err } -func (a *SQLiteAuditor) RecordWorkload(ctx context.Context, executionID int64, w WorkloadRecord) (int64, error) { - res, err := a.db.ExecContext(ctx, ` +func (a *SQLiteAuditor) RecordWorkload( + ctx context.Context, + executionID int64, + w WorkloadRecord, +) (int64, error) { + res, err := a.db.ExecContext( + ctx, ` INSERT INTO workload_details ( audit_id, workload_type, enabled, vm_count, cpu_cores, memory, has_data_disk, data_disk_size, requires_service, status @@ -193,8 +247,14 @@ func (a *SQLiteAuditor) UpdateWorkloadStatus(ctx context.Context, id int64, stat return err } -func (a *SQLiteAuditor) RecordVM(ctx context.Context, executionID int64, workloadID int64, v VMRecord) (int64, error) { - res, err := a.db.ExecContext(ctx, ` +func (a *SQLiteAuditor) RecordVM( + ctx context.Context, + executionID int64, + workloadID int64, + v VMRecord, +) (int64, error) { + res, err := a.db.ExecContext( + ctx, ` INSERT INTO vm_details ( audit_id, workload_id, vm_name, namespace, component, role, cpu_cores, memory, container_disk_image, has_data_disk, data_disk_size, @@ -210,7 +270,12 @@ func (a *SQLiteAuditor) RecordVM(ctx context.Context, executionID int64, workloa return res.LastInsertId() } -func (a *SQLiteAuditor) UpdateVMStatus(ctx context.Context, id int64, phase string, status string) error { +func (a *SQLiteAuditor) UpdateVMStatus( + ctx context.Context, + id int64, + phase string, + status string, +) error { if status == "ready" { _, err := a.db.ExecContext(ctx, `UPDATE vm_details SET phase = ?, status = ?, ready_at = ? WHERE id = ?`, @@ -230,8 +295,13 @@ func (a *SQLiteAuditor) RecordVMDeletion(ctx context.Context, id int64) error { return err } -func (a *SQLiteAuditor) RecordResource(ctx context.Context, executionID int64, r ResourceRecord) (int64, error) { - res, err := a.db.ExecContext(ctx, ` +func (a *SQLiteAuditor) RecordResource( + ctx context.Context, + executionID int64, + r ResourceRecord, +) (int64, error) { + res, err := a.db.ExecContext( + ctx, ` INSERT INTO resource_details (audit_id, resource_type, resource_name, namespace, status, created_at) VALUES (?, ?, ?, ?, 'created', ?)`, executionID, r.ResourceType, r.ResourceName, r.Namespace, now(), @@ -249,8 +319,82 @@ func (a *SQLiteAuditor) RecordResourceDeletion(ctx context.Context, id int64) er return err } +func (a *SQLiteAuditor) MarkVMsDeletedByName( + ctx context.Context, + vmNames []string, + namespace string, + runIDs []string, +) error { + if len(vmNames) == 0 || len(runIDs) == 0 { + return nil + } + query, args := buildMarkDeletedQuery( + "vm_details", "vm_name", "", + vmNames, namespace, runIDs, + ) + _, err := a.db.ExecContext(ctx, query, args...) + return err +} + +func (a *SQLiteAuditor) MarkResourcesDeletedByName( + ctx context.Context, + resourceType string, + names []string, + namespace string, + runIDs []string, +) error { + if len(names) == 0 || len(runIDs) == 0 { + return nil + } + query, args := buildMarkDeletedQuery( + "resource_details", "resource_name", resourceType, + names, namespace, runIDs, + ) + _, err := a.db.ExecContext(ctx, query, args...) + return err +} + +func buildMarkDeletedQuery( + table, nameCol, resourceType string, + names []string, + namespace string, + runIDs []string, +) (string, []interface{}) { + args := []interface{}{now()} + + namePlaceholders := make([]string, len(names)) + for i, n := range names { + namePlaceholders[i] = "?" + args = append(args, n) + } + args = append(args, namespace) + + runPlaceholders := make([]string, len(runIDs)) + for i, r := range runIDs { + runPlaceholders[i] = "?" + args = append(args, r) + } + + query := fmt.Sprintf( + `UPDATE %s SET status = 'deleted', deleted_at = ? + WHERE %s IN (%s) AND namespace = ? + AND audit_id IN (SELECT id FROM audit_log WHERE run_id IN (%s))`, + table, nameCol, + strings.Join(namePlaceholders, ","), + strings.Join(runPlaceholders, ","), + ) + + if resourceType != "" { + query += " AND resource_type = ?" + args = append(args, resourceType) + } + + return query, args +} + func (a *SQLiteAuditor) RecordEvent(ctx context.Context, executionID int64, e EventRecord) error { - _, err := a.db.ExecContext(ctx, ` + _, err := a.db.ExecContext( + ctx, ` INSERT INTO events (audit_id, vm_id, workload_id, event_type, message, error_detail, occurred_at) VALUES (?, ?, ?, ?, ?, ?, ?)`, executionID, e.VMID, e.WorkloadID, e.EventType, @@ -271,7 +415,11 @@ func (a *SQLiteAuditor) Close() error { // NoOpAuditor is an Auditor that does nothing, used when auditing is disabled. type NoOpAuditor struct{} -func (NoOpAuditor) StartExecution(_ context.Context, _ string, _ *config.Config) (int64, string, error) { +func (NoOpAuditor) StartExecution( + _ context.Context, + _ string, + _ *config.Config, +) (int64, string, error) { return 0, "", nil } @@ -280,7 +428,13 @@ func (NoOpAuditor) CompleteExecution(_ context.Context, _ int64, _ string, _ str } func (NoOpAuditor) LinkCleanupToRuns(_ context.Context, _ int64, _ []string) error { return nil } -func (NoOpAuditor) RecordCleanupCounts(_ context.Context, _ int64, _, _, _, _, _ int, _ bool) error { + +func (NoOpAuditor) RecordCleanupCounts( + _ context.Context, + _ int64, + _, _, _, _, _ int, + _ bool, +) error { return nil } @@ -291,12 +445,36 @@ func (NoOpAuditor) UpdateWorkloadStatus(_ context.Context, _ int64, _ string) er func (NoOpAuditor) RecordVM(_ context.Context, _ int64, _ int64, _ VMRecord) (int64, error) { return 0, nil } -func (NoOpAuditor) UpdateVMStatus(_ context.Context, _ int64, _ string, _ string) error { return nil } -func (NoOpAuditor) RecordVMDeletion(_ context.Context, _ int64) error { return nil } + +func (NoOpAuditor) UpdateVMStatus( + _ context.Context, + _ int64, + _ string, + _ string, +) error { + return nil +} + +func (NoOpAuditor) RecordVMDeletion(_ context.Context, _ int64) error { return nil } + func (NoOpAuditor) RecordResource(_ context.Context, _ int64, _ ResourceRecord) (int64, error) { return 0, nil } func (NoOpAuditor) RecordResourceDeletion(_ context.Context, _ int64) error { return nil } +func (NoOpAuditor) MarkVMsDeletedByName(_ context.Context, _ []string, _ string, _ []string) error { + return nil +} + +func (NoOpAuditor) MarkResourcesDeletedByName( + _ context.Context, + _ string, + _ []string, + _ string, + _ []string, +) error { + return nil +} + func (NoOpAuditor) RecordEvent(_ context.Context, _ int64, _ EventRecord) error { return nil } diff --git a/internal/audit/audit_test.go b/internal/audit/audit_test.go index d1934f4..022fd03 100644 --- a/internal/audit/audit_test.go +++ b/internal/audit/audit_test.go @@ -314,6 +314,149 @@ var _ = Describe("SQLiteAuditor", func() { }) }) + Describe("MarkVMsDeletedByName", func() { + It("marks matching VMs as deleted scoped by runID", func() { + cfg := &config.Config{Namespace: "test-ns"} + execID1, runID1, err := auditor.StartExecution(ctx, "run", cfg) + Expect(err).NotTo(HaveOccurred()) + + wlID1, err := auditor.RecordWorkload(ctx, execID1, audit.WorkloadRecord{ + WorkloadType: "cpu", Enabled: true, VMCount: 2, CPUCores: 2, Memory: "2Gi", + }) + Expect(err).NotTo(HaveOccurred()) + + vm1ID, err := auditor.RecordVM(ctx, execID1, wlID1, audit.VMRecord{ + VMName: "virtwork-cpu-0", Namespace: "test-ns", Component: "cpu", + CPUCores: 2, Memory: "2Gi", ContainerDiskImage: "img", + }) + Expect(err).NotTo(HaveOccurred()) + + vm2ID, err := auditor.RecordVM(ctx, execID1, wlID1, audit.VMRecord{ + VMName: "virtwork-cpu-1", Namespace: "test-ns", Component: "cpu", + CPUCores: 2, Memory: "2Gi", ContainerDiskImage: "img", + }) + Expect(err).NotTo(HaveOccurred()) + + // Create a second run with the same VM name — should NOT be affected + execID2, _, err := auditor.StartExecution(ctx, "run", cfg) + Expect(err).NotTo(HaveOccurred()) + + wlID2, err := auditor.RecordWorkload(ctx, execID2, audit.WorkloadRecord{ + WorkloadType: "cpu", Enabled: true, VMCount: 1, CPUCores: 2, Memory: "2Gi", + }) + Expect(err).NotTo(HaveOccurred()) + + otherRunVMID, err := auditor.RecordVM(ctx, execID2, wlID2, audit.VMRecord{ + VMName: "virtwork-cpu-0", Namespace: "test-ns", Component: "cpu", + CPUCores: 2, Memory: "2Gi", ContainerDiskImage: "img", + }) + Expect(err).NotTo(HaveOccurred()) + + Expect(auditor.MarkVMsDeletedByName(ctx, + []string{"virtwork-cpu-0", "virtwork-cpu-1"}, + "test-ns", + []string{runID1}, + )).To(Succeed()) + + db := auditor.DB() + + // Both VMs from run 1 should be deleted + for _, vmID := range []int64{vm1ID, vm2ID} { + var status string + var deletedAt sql.NullString + err = db.QueryRow(`SELECT status, deleted_at FROM vm_details WHERE id = ?`, vmID). + Scan(&status, &deletedAt) + Expect(err).NotTo(HaveOccurred()) + Expect(status).To(Equal("deleted")) + Expect(deletedAt.Valid).To(BeTrue()) + } + + // VM from run 2 should remain untouched + var otherStatus string + err = db.QueryRow(`SELECT status FROM vm_details WHERE id = ?`, otherRunVMID).Scan(&otherStatus) + Expect(err).NotTo(HaveOccurred()) + Expect(otherStatus).To(Equal("created")) + }) + + It("is a no-op when vmNames is empty", func() { + cfg := &config.Config{Namespace: "test-ns"} + _, runID, err := auditor.StartExecution(ctx, "run", cfg) + Expect(err).NotTo(HaveOccurred()) + + Expect(auditor.MarkVMsDeletedByName(ctx, []string{}, "test-ns", []string{runID})).To(Succeed()) + }) + + It("is a no-op when runIDs is empty", func() { + Expect(auditor.MarkVMsDeletedByName(ctx, []string{"virtwork-cpu-0"}, "test-ns", []string{})).To(Succeed()) + }) + }) + + Describe("MarkResourcesDeletedByName", func() { + It("marks matching resources as deleted scoped by type and runID", func() { + cfg := &config.Config{Namespace: "test-ns"} + execID1, runID1, err := auditor.StartExecution(ctx, "run", cfg) + Expect(err).NotTo(HaveOccurred()) + + secretID, err := auditor.RecordResource(ctx, execID1, audit.ResourceRecord{ + ResourceType: "Secret", ResourceName: "virtwork-cpu-0-cloudinit", Namespace: "test-ns", + }) + Expect(err).NotTo(HaveOccurred()) + + svcID, err := auditor.RecordResource(ctx, execID1, audit.ResourceRecord{ + ResourceType: "Service", ResourceName: "virtwork-iperf3-server", Namespace: "test-ns", + }) + Expect(err).NotTo(HaveOccurred()) + + // Second run — same secret name, should NOT be affected + execID2, _, err := auditor.StartExecution(ctx, "run", cfg) + Expect(err).NotTo(HaveOccurred()) + + otherSecretID, err := auditor.RecordResource(ctx, execID2, audit.ResourceRecord{ + ResourceType: "Secret", ResourceName: "virtwork-cpu-0-cloudinit", Namespace: "test-ns", + }) + Expect(err).NotTo(HaveOccurred()) + + // Mark only Secrets from run 1 as deleted + Expect(auditor.MarkResourcesDeletedByName(ctx, + "Secret", + []string{"virtwork-cpu-0-cloudinit"}, + "test-ns", + []string{runID1}, + )).To(Succeed()) + + db := auditor.DB() + + // Secret from run 1 should be deleted + var status string + var deletedAt sql.NullString + err = db.QueryRow(`SELECT status, deleted_at FROM resource_details WHERE id = ?`, secretID). + Scan(&status, &deletedAt) + Expect(err).NotTo(HaveOccurred()) + Expect(status).To(Equal("deleted")) + Expect(deletedAt.Valid).To(BeTrue()) + + // Service from run 1 should NOT be affected (different type) + err = db.QueryRow(`SELECT status FROM resource_details WHERE id = ?`, svcID).Scan(&status) + Expect(err).NotTo(HaveOccurred()) + Expect(status).To(Equal("created")) + + // Secret from run 2 should NOT be affected + err = db.QueryRow(`SELECT status FROM resource_details WHERE id = ?`, otherSecretID).Scan(&status) + Expect(err).NotTo(HaveOccurred()) + Expect(status).To(Equal("created")) + }) + + It("is a no-op when names is empty", func() { + cfg := &config.Config{Namespace: "test-ns"} + _, runID, err := auditor.StartExecution(ctx, "run", cfg) + Expect(err).NotTo(HaveOccurred()) + + Expect(auditor.MarkResourcesDeletedByName( + ctx, "Secret", []string{}, "test-ns", []string{runID}), + ).To(Succeed()) + }) + }) + Describe("event recording", func() { It("records events with optional VM and workload IDs", func() { cfg := &config.Config{Namespace: "test-ns"} @@ -560,6 +703,9 @@ var _ = Describe("NoOpAuditor", func() { Expect(resID).To(Equal(int64(0))) Expect(a.RecordResourceDeletion(ctx, 0)).To(Succeed()) + Expect(a.MarkVMsDeletedByName(ctx, []string{"vm-0"}, "ns", []string{"run-1"})).To(Succeed()) + Expect(a.MarkResourcesDeletedByName(ctx, "Secret", []string{"s-0"}, "ns", []string{"run-1"})).To(Succeed()) + Expect(a.RecordEvent(ctx, 0, audit.EventRecord{})).To(Succeed()) Expect(a.Close()).To(Succeed()) }) diff --git a/internal/cleanup/cleanup.go b/internal/cleanup/cleanup.go index 5137812..26eed5e 100644 --- a/internal/cleanup/cleanup.go +++ b/internal/cleanup/cleanup.go @@ -28,6 +28,12 @@ type CleanupResult struct { NamespaceDeleted bool Errors []error RunIDs []string // unique run IDs collected from cleaned-up resources + + DeletedVMNames []string + DeletedServiceNames []string + DeletedSecretNames []string + DeletedDVNames []string + DeletedPVCNames []string } // CleanupPreview summarises what a cleanup operation would delete, without modifying anything. @@ -163,6 +169,7 @@ func CleanupAll( continue } result.VMsDeleted++ + result.DeletedVMNames = append(result.DeletedVMNames, vmList.Items[i].Name) } // Delete services by label @@ -179,6 +186,7 @@ func CleanupAll( continue } result.ServicesDeleted++ + result.DeletedServiceNames = append(result.DeletedServiceNames, svcList.Items[i].Name) } // Delete secrets by label @@ -198,6 +206,7 @@ func CleanupAll( continue } result.SecretsDeleted++ + result.DeletedSecretNames = append(result.DeletedSecretNames, secretList.Items[i].Name) } // Delete DataVolumes by label (before PVCs — DV controller may GC owned PVCs) @@ -216,6 +225,7 @@ func CleanupAll( continue } result.DVsDeleted++ + result.DeletedDVNames = append(result.DeletedDVNames, dvList.Items[i].Name) } // Delete PVCs by label @@ -232,6 +242,7 @@ func CleanupAll( continue } result.PVCsDeleted++ + result.DeletedPVCNames = append(result.DeletedPVCNames, pvcList.Items[i].Name) } // Collect unique run IDs diff --git a/internal/cleanup/cleanup_test.go b/internal/cleanup/cleanup_test.go index faa9bce..db9a04f 100644 --- a/internal/cleanup/cleanup_test.go +++ b/internal/cleanup/cleanup_test.go @@ -686,6 +686,49 @@ var _ = Describe("CleanupAll", func() { Expect(result.Errors).To(BeEmpty()) }) + It("should track deleted resource names", func() { + vm1 := newManagedVM("vm-1") + vm2 := newManagedVM("vm-2") + svc1 := newManagedService("svc-1") + sec1 := newManagedSecret("sec-1") + c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(vm1, vm2, svc1, sec1).Build() + + result, err := cleanup.CleanupAll(ctx, c, &config.Config{Namespace: namespace}, false, "") + Expect(err).NotTo(HaveOccurred()) + Expect(result.DeletedVMNames).To(ConsistOf("vm-1", "vm-2")) + Expect(result.DeletedServiceNames).To(ConsistOf("svc-1")) + Expect(result.DeletedSecretNames).To(ConsistOf("sec-1")) + Expect(result.DeletedDVNames).To(BeEmpty()) + Expect(result.DeletedPVCNames).To(BeEmpty()) + }) + + // nolint: dupl + It("should not include failed deletions in name lists", func() { + vm1 := newManagedVM("vm-1") + vm2 := newManagedVM("vm-2") + callCount := 0 + c := fake.NewClientBuilder(). + WithScheme(scheme). + WithObjects(vm1, vm2). + WithInterceptorFuncs(interceptor.Funcs{ + Delete: func(ctx context.Context, cl client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + if _, ok := obj.(*kubevirtv1.VirtualMachine); ok { + callCount++ + if callCount == 1 { + return fmt.Errorf("VM deletion; %w", ErrDeleteResource) + } + } + return cl.Delete(ctx, obj, opts...) + }, + }). + Build() + + result, err := cleanup.CleanupAll(ctx, c, &config.Config{Namespace: namespace}, false, "") + Expect(err).NotTo(HaveOccurred()) + Expect(result.DeletedVMNames).To(HaveLen(1)) + Expect(result.VMsDeleted).To(Equal(1)) + }) + It("should handle empty namespace gracefully", func() { c := fake.NewClientBuilder().WithScheme(scheme).Build() diff --git a/internal/orchestrator/cleanup.go b/internal/orchestrator/cleanup.go index acf27fd..ac7954f 100644 --- a/internal/orchestrator/cleanup.go +++ b/internal/orchestrator/cleanup.go @@ -67,9 +67,17 @@ func (co *CleanupOrchestrator) Execute( ) (*cleanup.CleanupResult, error) { if err := co.auditor.RecordEvent(ctx, execID, audit.EventRecord{ EventType: "cleanup_started", - Message: fmt.Sprintf("Starting cleanup (namespace: %s, run-id filter: %q)", co.config.Namespace, runID), + Message: fmt.Sprintf( + "Starting cleanup (namespace: %s, run-id filter: %q)", + co.config.Namespace, + runID, + ), }); err != nil { - co.logger.Warn("audit record failed", slog.String("op", "RecordEvent"), slog.String("error", err.Error())) + co.logger.Warn( + "audit record failed", + slog.String("op", "RecordEvent"), + slog.String("error", err.Error()), + ) } result, err := cleanup.CleanupAll(ctx, co.client, co.config, deleteNamespace, runID) @@ -87,6 +95,8 @@ func (co *CleanupOrchestrator) Execute( } } + co.markDeletionsInAudit(ctx, result) + if auditErr := co.auditor.RecordCleanupCounts(ctx, execID, result.VMsDeleted, result.ServicesDeleted, result.SecretsDeleted, result.DVsDeleted, result.PVCsDeleted, result.NamespaceDeleted); auditErr != nil { @@ -103,8 +113,61 @@ func (co *CleanupOrchestrator) Execute( result.VMsDeleted, result.ServicesDeleted, result.SecretsDeleted, result.DVsDeleted, result.PVCsDeleted), }); auditErr != nil { - co.logger.Warn("audit record failed", slog.String("op", "RecordEvent"), slog.String("error", auditErr.Error())) + co.logger.Warn( + "audit record failed", + slog.String("op", "RecordEvent"), + slog.String("error", auditErr.Error()), + ) } return result, nil } + +func (co *CleanupOrchestrator) markDeletionsInAudit( + ctx context.Context, + result *cleanup.CleanupResult, +) { + ns := co.config.Namespace + + if len(result.DeletedVMNames) > 0 && len(result.RunIDs) > 0 { + if err := co.auditor.MarkVMsDeletedByName( + ctx, + result.DeletedVMNames, + ns, + result.RunIDs, + ); err != nil { + co.logger.Warn( + "audit record failed", + slog.String("op", "MarkVMsDeletedByName"), + slog.String("error", err.Error()), + ) + } + } + + for _, rt := range []struct { + typ string + names []string + }{ + {"Service", result.DeletedServiceNames}, + {"Secret", result.DeletedSecretNames}, + {"DataVolume", result.DeletedDVNames}, + {"PersistentVolumeClaim", result.DeletedPVCNames}, + } { + if len(rt.names) > 0 && len(result.RunIDs) > 0 { + if err := co.auditor.MarkResourcesDeletedByName( + ctx, + rt.typ, + rt.names, + ns, + result.RunIDs, + ); err != nil { + co.logger.Warn( + "audit record failed", + slog.String("op", "MarkResourcesDeletedByName"), + slog.String("type", rt.typ), + slog.String("error", err.Error()), + ) + } + } + } +} diff --git a/internal/orchestrator/cleanup_test.go b/internal/orchestrator/cleanup_test.go index b30c0f8..458531e 100644 --- a/internal/orchestrator/cleanup_test.go +++ b/internal/orchestrator/cleanup_test.go @@ -179,5 +179,87 @@ var _ = Describe("CleanupOrchestrator", func() { Expect(err).NotTo(HaveOccurred()) Expect(result).NotTo(BeNil()) }) + + It("should mark VMs and resources as deleted in audit", func() { + sqlAuditor, err := audit.NewSQLiteAuditor(":memory:") + Expect(err).NotTo(HaveOccurred()) + defer func() { _ = sqlAuditor.Close() }() + + runCfg := &config.Config{Namespace: constants.DefaultNamespace} + runExecID, runID, err := sqlAuditor.StartExecution(ctx, "run", runCfg) + Expect(err).NotTo(HaveOccurred()) + + wlID, err := sqlAuditor.RecordWorkload(ctx, runExecID, audit.WorkloadRecord{ + WorkloadType: "cpu", Enabled: true, VMCount: 1, CPUCores: 2, Memory: "2Gi", + }) + Expect(err).NotTo(HaveOccurred()) + + _, err = sqlAuditor.RecordVM(ctx, runExecID, wlID, audit.VMRecord{ + VMName: "virtwork-cpu-0", Namespace: constants.DefaultNamespace, Component: "cpu", + CPUCores: 2, Memory: "2Gi", ContainerDiskImage: "img", + }) + Expect(err).NotTo(HaveOccurred()) + + _, err = sqlAuditor.RecordResource(ctx, runExecID, audit.ResourceRecord{ + ResourceType: "Secret", ResourceName: "virtwork-cpu-0-cloudinit", + Namespace: constants.DefaultNamespace, + }) + Expect(err).NotTo(HaveOccurred()) + + Expect(sqlAuditor.CompleteExecution(ctx, runExecID, "success", "")).To(Succeed()) + + // Create matching K8s resources with the run-id label + vm := &kubevirtv1.VirtualMachine{ + ObjectMeta: metav1.ObjectMeta{ + Name: "virtwork-cpu-0", + Namespace: constants.DefaultNamespace, + Labels: map[string]string{ + constants.LabelManagedBy: constants.ManagedByValue, + constants.LabelRunID: runID, + }, + }, + } + secret := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: "virtwork-cpu-0-cloudinit", + Namespace: constants.DefaultNamespace, + Labels: map[string]string{ + constants.LabelManagedBy: constants.ManagedByValue, + constants.LabelRunID: runID, + }, + }, + } + ns := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: constants.DefaultNamespace}, + } + c := newFakeClient(ns, vm, secret) + logger := logging.NewLogger(buf, false) + co := orchestrator.NewCleanupOrchestrator(logger, c, cfg, sqlAuditor, buf) + + cleanupExecID, _, err := sqlAuditor.StartExecution(ctx, "cleanup", runCfg) + Expect(err).NotTo(HaveOccurred()) + + result, err := co.Execute(ctx, cleanupExecID, false, "") + Expect(err).NotTo(HaveOccurred()) + Expect(result.VMsDeleted).To(Equal(1)) + Expect(result.SecretsDeleted).To(Equal(1)) + + db := sqlAuditor.DB() + var vmStatus string + err = db.QueryRow( + `SELECT status FROM vm_details WHERE vm_name = ? AND namespace = ?`, + "virtwork-cpu-0", constants.DefaultNamespace, + ).Scan(&vmStatus) + Expect(err).NotTo(HaveOccurred()) + Expect(vmStatus).To(Equal("deleted")) + + var resStatus string + err = db.QueryRow( + `SELECT status FROM resource_details WHERE resource_name = ? AND namespace = ?`, + "virtwork-cpu-0-cloudinit", constants.DefaultNamespace, + ).Scan(&resStatus) + Expect(err).NotTo(HaveOccurred()) + Expect(resStatus).To(Equal("deleted")) + }) }) }) From e8e287bf355a0b581ac44a6f01bcb51833dfd071 Mon Sep 17 00:00:00 2001 From: Melvin Hillsman Date: Wed, 10 Jun 2026 21:31:08 -0500 Subject: [PATCH 4/4] docs: add mermaid sequence diagrams Signed-off-by: Melvin Hillsman --- docs/mermaid/00-high-level-overview.mermaid | 88 ++++++ docs/mermaid/01-workload-development.mermaid | 128 +++++++++ docs/mermaid/02-contributor-deep-dive.mermaid | 264 ++++++++++++++++++ .../03-orchestrator-run-phases.mermaid | 191 +++++++++++++ docs/mermaid/04-config-resolution.mermaid | 106 +++++++ docs/mermaid/05-vm-lifecycle.mermaid | 152 ++++++++++ docs/mermaid/06-audit-lifecycle.mermaid | 183 ++++++++++++ docs/mermaid/07-cloudinit-generation.mermaid | 124 ++++++++ docs/mermaid/08-cleanup-mechanics.mermaid | 198 +++++++++++++ 9 files changed, 1434 insertions(+) create mode 100644 docs/mermaid/00-high-level-overview.mermaid create mode 100644 docs/mermaid/01-workload-development.mermaid create mode 100644 docs/mermaid/02-contributor-deep-dive.mermaid create mode 100644 docs/mermaid/03-orchestrator-run-phases.mermaid create mode 100644 docs/mermaid/04-config-resolution.mermaid create mode 100644 docs/mermaid/05-vm-lifecycle.mermaid create mode 100644 docs/mermaid/06-audit-lifecycle.mermaid create mode 100644 docs/mermaid/07-cloudinit-generation.mermaid create mode 100644 docs/mermaid/08-cleanup-mechanics.mermaid diff --git a/docs/mermaid/00-high-level-overview.mermaid b/docs/mermaid/00-high-level-overview.mermaid new file mode 100644 index 0000000..6caefd2 --- /dev/null +++ b/docs/mermaid/00-high-level-overview.mermaid @@ -0,0 +1,88 @@ +%% High-Level Overview: virtwork run & cleanup flows +%% Audience: evaluators, architects, new team members onboarding + +sequenceDiagram + autonumber + + actor User + participant CLI as virtwork CLI + participant Config as Configuration + participant Auditor as Audit (SQLite) + participant Orchestrator as Orchestrator + participant Cluster as OpenShift Cluster + + %% ─── RUN FLOW ──────────────────────────────────────────────── + + Note over User, Cluster: virtwork run + + User ->> CLI: virtwork run --workloads cpu,network ... + CLI ->> Config: Load config (flags > env > file > defaults) + Config -->> CLI: Config with workloads, VM specs, SSH creds + CLI ->> CLI: Validate workload names against registry + CLI ->> Auditor: Open database, run migrations + Auditor -->> CLI: Auditor ready + + CLI ->> Cluster: Connect via kubeconfig + Cluster -->> CLI: Kubernetes client + + CLI ->> Auditor: StartExecution(command, config) + Auditor -->> CLI: executionID, runID (UUID) + + CLI ->> Orchestrator: Run(workloads, runID) + + Note over Orchestrator, Cluster: Orchestration phases + Orchestrator ->> Orchestrator: Plan VMs (resolve workloads, build specs) + Orchestrator ->> Cluster: Ensure namespace exists + Orchestrator ->> Cluster: Create services (for multi-VM workloads) + Orchestrator ->> Cluster: Create cloud-init secrets (parallel) + Orchestrator ->> Cluster: Create VMs (parallel, concurrency-limited) + Orchestrator ->> Cluster: Wait for VMs to reach Running (parallel polling) + + Orchestrator -->> CLI: RunResult (counts: VMs, services, secrets) + CLI ->> Auditor: CompleteExecution(status=success) + CLI -->> User: Summary (created N VMs, M services, ...) + + %% ─── CLEANUP FLOW ──────────────────────────────────────────── + + Note over User, Cluster: virtwork cleanup + + User ->> CLI: virtwork cleanup [--run-id UUID] + CLI ->> Config: Load config + CLI ->> Auditor: Open database, run migrations + Auditor -->> CLI: Auditor ready + CLI ->> Cluster: Connect via kubeconfig + Cluster -->> CLI: Kubernetes client + CLI ->> Auditor: StartExecution(command=cleanup) + Auditor -->> CLI: executionID + + CLI ->> Orchestrator: Preview(runID filter) + Orchestrator ->> Cluster: List resources by label selector + Note right of Cluster: Label: app.kubernetes.io/managed-by=virtwork
Optional: virtwork/run-id=UUID + Cluster -->> Orchestrator: Resource counts + Orchestrator -->> CLI: CleanupPreview + + CLI -->> User: Show preview (N VMs, M services, ...) + User ->> CLI: Confirm deletion (--run-id or --yes skips prompt) + + CLI ->> Orchestrator: Execute cleanup + Orchestrator ->> Cluster: Delete VMs + Orchestrator ->> Cluster: Delete Services + Orchestrator ->> Cluster: Delete Secrets + Orchestrator ->> Cluster: Delete DataVolumes + Orchestrator ->> Cluster: Delete PVCs + opt Namespace deletion requested + Orchestrator ->> Cluster: Delete Namespace + end + Orchestrator ->> Auditor: LinkCleanupToRuns(collected runIDs) + Orchestrator ->> Auditor: RecordCleanupCounts(deletion totals) + Orchestrator -->> CLI: CleanupResult (deletion counts, collected runIDs) + + CLI ->> Auditor: CompleteExecution(status=success) + CLI -->> User: Cleanup summary + + %% ─── AUDIT CROSS-CUTTING ──────────────────────────────────── + + Note over Auditor: Audit records (SQLite, optional via --no-audit) + Note over Auditor: Tables: audit_log, workload_details,
vm_details, resource_details, events + Note over Auditor: Each run gets a UUID (virtwork/run-id label)
applied to all K8s resources for traceability + Note over Auditor: NoOpAuditor used when auditing is disabled diff --git a/docs/mermaid/01-workload-development.mermaid b/docs/mermaid/01-workload-development.mermaid new file mode 100644 index 0000000..31691a7 --- /dev/null +++ b/docs/mermaid/01-workload-development.mermaid @@ -0,0 +1,128 @@ +%% Workload Development: using, creating, and extending workloads +%% Audience: users running existing workloads, developers adding new workload types + +sequenceDiagram + autonumber + + actor User + participant CLI as virtwork CLI + participant Config as Config / Viper + participant Registry as Workload Registry + participant Workload as Workload Instance + participant CloudInit as CloudInit Builder + participant Orch as Orchestrator + participant Cluster as OpenShift Cluster + + %% ─── WORKLOAD RESOLUTION ───────────────────────────────────── + + Note over User, Registry: Workload resolution from CLI to concrete instance + User ->> CLI: --workloads cpu,network,disk + CLI ->> CLI: ValidateWorkloadNames(["cpu","network","disk"]) + Note right of CLI: Checks names against DefaultRegistry
Suggests fixes via Levenshtein distance
e.g. "cpuu" → "did you mean cpu?" + + CLI ->> Config: LoadConfig(cmd) — merge flag+env+file+defaults + Note right of Config: Per-workload overrides from config.yaml:
workloads:
cpu:
vm_count: 3
cpu_cores: 4
memory: "4Gi" + Config -->> CLI: Config with map[string]WorkloadConfig + + %% ─── SINGLE-VM WORKLOAD (e.g. CPU) ────────────────────────── + + Note over Registry, Cluster: Single-VM workload flow (e.g. cpu, memory, disk, chaos-*) + CLI ->> Orch: Run(workloadNames, runID) + Orch ->> Registry: DefaultRegistry() + Registry -->> Orch: Registry map[name] → factory + + Orch ->> Registry: Get("cpu", workloadConfig, opts...) + Registry -->> Orch: *CPUWorkload (implements Workload) + + Orch ->> Workload: VMCount() + Workload -->> Orch: N (from config, default 1) + + Orch ->> Workload: VMResources() + Workload -->> Orch: {CPUCores, Memory} + + Orch ->> Workload: CloudInitUserdata() + Workload ->> CloudInit: BuildCloudConfig(opts) + Note right of CloudInit: Packages: ["stress-ng"]
WriteFiles: [systemd unit]
RunCmd: [enable service]
SSH credentials injected + CloudInit -->> Workload: "#cloud-config\n..." + Workload -->> Orch: cloud-init YAML + + Orch ->> Workload: DataVolumeTemplates() + Workload -->> Orch: nil (CPU needs no data disk) + + Orch ->> Workload: RequiresService() + Workload -->> Orch: false + + loop For each VM (0..N-1) + Note over Orch: VM name: virtwork-cpu-0, virtwork-cpu-1, ... + Orch ->> Cluster: Create cloud-init Secret + Orch ->> Cluster: Create VirtualMachine + end + + %% ─── MULTI-VM WORKLOAD (e.g. network) ─────────────────────── + + Note over Registry, Cluster: Multi-VM workload flow (e.g. network, tps) + Orch ->> Registry: Get("network", workloadConfig, opts...) + Registry -->> Orch: *NetworkWorkload (implements MultiVMWorkload) + + Orch ->> Workload: RoleDistribution() + Workload -->> Orch: [{server, N}, {client, N}] + + Orch ->> Workload: RequiresService() + Workload -->> Orch: true + + Orch ->> Workload: ServiceSpec() + Workload -->> Orch: ClusterIP Service (iperf3 port 5201) + Orch ->> Cluster: Create Service (virtwork-iperf3-server) + + loop For each role in RoleDistribution + Orch ->> Workload: UserdataForRole(role, namespace) + alt role = "server" + Workload ->> CloudInit: BuildCloudConfig(iperf3 -s) + else role = "client" + Workload ->> CloudInit: BuildCloudConfig(iperf3 -c server.ns.svc) + end + CloudInit -->> Workload: role-specific cloud-init YAML + Workload -->> Orch: cloud-init YAML + + loop For each VM in role (0..perRole-1) + Note over Orch: VM name: virtwork-network-server-0, ...-client-0 + Orch ->> Cluster: Create cloud-init Secret + Orch ->> Cluster: Create VirtualMachine + end + end + + %% ─── WORKLOAD WITH DATA DISK (e.g. disk, database) ────────── + + Note over Registry, Cluster: Workload with persistent storage (e.g. disk, database, chaos-disk) + Orch ->> Registry: Get("disk", workloadConfig, opts...) + Registry -->> Orch: *DiskWorkload + + Orch ->> Workload: DataVolumeTemplates() + Workload -->> Orch: DataVolumeTemplateSpec (blank PVC, e.g. 20Gi) + + Orch ->> Workload: ExtraDisks() + Workload -->> Orch: Disk{Name: "data-disk", Bus: virtio} + + Orch ->> Workload: ExtraVolumes() + Workload -->> Orch: Volume{DataVolume source} + + Note over Orch: NamespaceDataVolumes() uniquifies disk names per VM:
data-disk → virtwork-disk-0-data-disk + + loop For each VM + Orch ->> Cluster: Create VM with DataVolumeTemplate + Note right of Cluster: CDI provisions PVC automatically
from DataVolumeTemplate + end + + %% ─── CREATING A NEW WORKLOAD ───────────────────────────────── + + Note over User, Registry: How to add a new workload type + + Note over Workload: Implement the Workload interface:
─ Name() string
─ CloudInitUserdata() (string, error)
─ VMResources() VMResourceSpec
─ ExtraVolumes() []Volume
─ ExtraDisks() []Disk
─ DataVolumeTemplates() ([]DVT, error)
─ RequiresService() bool
─ ServiceSpec() *Service
─ VMCount() int + + Note over Workload: For multi-VM: also implement MultiVMWorkload:
─ RoleDistribution() []RoleSpec
─ UserdataForRole(role, namespace) (string, error) + + Note over Workload: Embed BaseWorkload for defaults:
─ VMResources, VMCount, ExtraVolumes,
ExtraDisks, DataVolumeTemplates,
RequiresService, ServiceSpec all have
sensible default implementations + + Note over Registry: Register in DefaultRegistry():
"my-workload": func(cfg, opts) Workload {
return NewMyWorkload(cfg, opts.SSHUser, ...)
} + + Note over CloudInit: Build cloud-init via BaseWorkload.BuildCloudConfig:
─ Specify packages to install
─ Write systemd unit files
─ Run commands to enable services
─ SSH credentials are injected automatically diff --git a/docs/mermaid/02-contributor-deep-dive.mermaid b/docs/mermaid/02-contributor-deep-dive.mermaid new file mode 100644 index 0000000..098d116 --- /dev/null +++ b/docs/mermaid/02-contributor-deep-dive.mermaid @@ -0,0 +1,264 @@ +%% Contributor Deep-Dive: internal mechanics, concurrency, audit recording, cleanup +%% Audience: contributors resolving tech debt, fixing bugs, building novel workloads + +sequenceDiagram + autonumber + + participant CLI as cmd/virtwork
(runE / cleanupE) + participant RO as RunOrchestrator + participant Reg as Registry + participant WL as Workload + participant Res as resources pkg + participant VM as vm pkg + participant Wait as wait pkg + participant Aud as SQLiteAuditor + participant DB as SQLite DB + participant CO as CleanupOrchestrator + participant CL as cleanup pkg + participant K8s as K8s API + + %% ─── INITIALIZATION ───────────────────────────────────────── + + Note over CLI, DB: Initialization (runE entry point) + CLI ->> CLI: config.LoadConfig(cmd)
flags > env > config.yaml > defaults + CLI ->> CLI: workloads.ValidateWorkloadNames(names) + CLI ->> CLI: logging.NewLogger(writer, verbose) + + CLI ->> Aud: NewSQLiteAuditor(dbPath) + Aud ->> DB: Open DB with WAL + foreign keys + Aud ->> DB: migrateDB() + Note right of DB: 1. Create schema_version table
2. Read current version
3. Apply pending migrations in tx
4. migrateV1Baseline executes SchemaSQL + DB -->> Aud: Migrated + + CLI ->> K8s: cluster.Connect(kubeconfigPath) + K8s -->> CLI: client.Client (controller-runtime) + + CLI ->> Aud: StartExecution(ctx, "run", cfg) + Aud ->> DB: INSERT audit_log (run_id=uuid, command, config snapshot) + DB -->> Aud: executionID, runID + Aud -->> CLI: executionID, runID + + %% ─── PLAN VMS ──────────────────────────────────────────────── + + Note over RO, WL: Phase 1: planVMs — resolve workloads, build VM specs + + CLI ->> RO: NewRunOrchestrator(logger, client, cfg, auditor, writer) + CLI ->> RO: Run(ctx, execID, runID, workloadNames, vmCountFlag) + RO ->> Reg: DefaultRegistry() + Note right of Reg: Pre-populated map:
cpu, memory, disk, database,
network, tps, chaos-disk,
chaos-network, chaos-process + + loop For each workload name + RO ->> RO: Check if disabled in config file + alt Workload disabled + RO ->> Aud: RecordEvent(execID, "workload_skipped") + else Workload enabled + RO ->> Reg: Get(name, workloadConfig, opts...) + Reg ->> Reg: Resolve WorkloadFactory + Reg ->> Reg: Build RegistryOpts from functional options
(WithNamespace, WithSSHCredentials, WithDataDiskSize) + Reg ->> WL: factory(cfg, registryOpts) + WL -->> Reg: Workload instance + Reg -->> RO: Workload + + RO ->> Aud: RecordWorkload(execID, WorkloadRecord) + Aud ->> DB: INSERT workload_details + DB -->> Aud: workloadID + Aud -->> RO: workloadID + + RO ->> WL: VMCount() / RoleDistribution() + RO ->> WL: VMResources() → {CPUCores, Memory} + RO ->> WL: DataVolumeTemplates() + + alt Workload implements MultiVMWorkload + loop For each RoleSpec in RoleDistribution() + RO ->> WL: UserdataForRole(role, namespace) + WL -->> RO: role-specific cloud-init YAML + loop For each VM in role (0-indexed) + RO ->> RO: Build VMPlan{WorkloadName, VMName, Component, Role, VMSpecInput} + Note right of RO: VM name: virtwork-{name}-{role}-{i}
Labels: managed-by, component, run-id, role + end + end + else Single-VM Workload + RO ->> WL: CloudInitUserdata() + WL -->> RO: cloud-init YAML + loop For each VM (0..VMCount-1) + RO ->> RO: Build VMPlan{VMSpecInput} + Note right of RO: VM name: virtwork-{name}-{i}
NamespaceDataVolumes() uniquifies DVT names + end + end + end + end + + RO -->> RO: Returns: []VMPlan, []vmNames,
map[name]Workload, map[name]auditWorkloadID + + %% ─── CREATE RESOURCES ──────────────────────────────────────── + + Note over RO, K8s: Phase 2: createResources — namespace + services + RO ->> Res: EnsureNamespace(ctx, client, namespace, labels) + Res ->> K8s: Create Namespace (idempotent: AlreadyExists → merge labels) + K8s -->> Res: OK + + loop For each workload requiring service + RO ->> WL: ServiceSpec() + WL -->> RO: *corev1.Service + RO ->> Res: CreateService(ctx, client, svc) + Res ->> K8s: Create Service + K8s -->> Res: OK + RO ->> Aud: RecordResource(execID, ResourceRecord{type, name, ns}) + Aud ->> DB: INSERT resource_details + RO ->> Aud: RecordEvent(execID, "service_created") + Aud ->> DB: INSERT events + end + + %% ─── CREATE SECRETS (PARALLEL) ────────────────────────────── + + Note over RO, K8s: Phase 3: createSecrets — errgroup with limit 10 + Note right of RO: errgroup.WithContext(ctx)
g.SetLimit(10) + + par Concurrent secret creation (up to 10 at a time) + loop For each VMPlan + RO ->> Res: CreateCloudInitSecret(ctx, client, name, ns, userdata, labels) + Note right of Res: Secret name: {vmName}-cloudinit
Validates size ≤ 1 MiB
StringData["userdata"] = cloud-init YAML + Res ->> K8s: Create Secret + K8s -->> Res: OK + RO ->> RO: plan.VMSpec.CloudInitSecretName = secretName + RO ->> Aud: RecordResource(execID, ResourceRecord{type: "Secret"}) + Aud ->> DB: INSERT resource_details + end + end + + %% ─── CREATE VMS (PARALLEL) ────────────────────────────────── + + Note over RO, K8s: Phase 4: createVMs — errgroup with VMConcurrency limit + Note right of RO: errgroup.WithContext(ctx)
g.SetLimit(cfg.VMConcurrency) + + par Concurrent VM creation (up to VMConcurrency at a time) + loop For each VMPlan + RO ->> VM: BuildVMSpec(VMSpecOpts{...}) + Note right of VM: Constructs kubevirtv1.VirtualMachine:
─ RunStrategy: Always
─ Disks: container + cloudInit + extras
─ Volumes: containerDisk + secretRef + extras
─ Network: masquerade + pod
─ DataVolumeTemplates for persistent storage + VM -->> RO: *kubevirtv1.VirtualMachine + + RO ->> VM: CreateVM(ctx, client, vmObj) + VM ->> K8s: client.Create(ctx, vm) + Note right of K8s: retryOnTransient: 1 + 5 retries
exponential backoff on
TooManyRequests / server errors
AlreadyExists → success (idempotent) + K8s -->> VM: OK + VM -->> RO: nil + + RO ->> Aud: RecordVM(execID, workloadID, VMRecord) + Aud ->> DB: INSERT vm_details + RO ->> Aud: RecordEvent(execID, EventRecord{type: "vm_created"}) + Aud ->> DB: INSERT events + end + end + + %% ─── WAIT FOR READINESS (PARALLEL) ────────────────────────── + + Note over RO, K8s: Phase 5: waitForReadiness — WaitGroup + goroutine per VM + alt cfg.WaitForReady == true + RO ->> Wait: WaitForAllVMsReady(ctx, client, logger, vmNames, ns, timeout, interval) + + Note right of Wait: sync.WaitGroup + sync.Mutex
one goroutine per VM (unbounded) + + par One goroutine per VM + loop Poll until Running or timeout + Wait ->> K8s: Get VirtualMachineInstance + K8s -->> Wait: VMI status + alt vmi.Status.Phase == Running + Wait -->> Wait: VM ready + else Not ready + Wait ->> Wait: Sleep(pollInterval) + end + end + end + + Wait -->> RO: map[vmName]error + + loop For each VM result + alt VM ready + RO ->> Aud: RecordEvent(execID, "vm_ready") + else VM timed out + RO ->> Aud: RecordEvent(execID, "vm_timeout") + end + end + + alt Any failures + loop For each workload + alt Has failed VMs + RO ->> Aud: UpdateWorkloadStatus(workloadID, "failed") + else All VMs ready + RO ->> Aud: UpdateWorkloadStatus(workloadID, "created") + end + end + RO -->> RO: Return error (ErrReadinessCheck) + end + else cfg.WaitForReady == false + Note over RO: Skip readiness check (--no-wait) + end + + %% ─── COMPLETION ───────────────────────────────────────────── + + RO -->> CLI: RunResult{RunID, VMCount, ServiceCount, SecretCount} + CLI ->> Aud: CompleteExecution(execID, "success", "") + Aud ->> DB: UPDATE audit_log SET status, completed_at + CLI ->> CLI: printSummary(result) + CLI -->> CLI: Return nil + + %% ═══ CLEANUP DEEP-DIVE ══════════════════════════════════════ + + Note over CLI, K8s: Cleanup flow (cleanupE entry point) + CLI ->> CO: NewCleanupOrchestrator(logger, client, cfg, auditor, writer) + + Note over CO, K8s: Preview phase — read-only label query + CLI ->> CO: Preview(ctx, execID, targetRunID) + CO ->> CL: PreviewCleanup(ctx, client, config, runID) + CL ->> K8s: List VMs (label: managed-by=virtwork [+ run-id=UUID]) + CL ->> K8s: List Services (same labels) + CL ->> K8s: List Secrets (same labels) + CL ->> K8s: List DataVolumes (same labels) + CL ->> K8s: List PVCs (same labels) + CL ->> CL: collectRunID() from each resource's labels + CL -->> CO: CleanupPreview{counts, runIDs} + CO -->> CLI: CleanupPreview + + CLI ->> CLI: printCleanupPreview() + CLI -->> CLI: PromptForConfirmation() (unless --run-id or --yes set) + + Note over CO, K8s: Execute phase — ordered deletion + CLI ->> CO: Execute(ctx, execID, deleteNS, targetRunID) + CO ->> Aud: RecordEvent(execID, "cleanup_started") + + CO ->> CL: CleanupAll(ctx, client, config, deleteNS, runID) + + Note right of CL: Deletion order matters:
VMs first (stop workloads)
Services (remove networking)
Secrets (remove config)
DataVolumes (release storage)
PVCs (release claims)
Namespace last (if requested) + + loop For each resource type (VMs → SVCs → Secrets → DVs → PVCs) + CL ->> K8s: List resources by label + loop For each resource + CL ->> CL: collectRunID(labels) + CL ->> K8s: Delete resource + Note right of K8s: NotFound → skip (idempotent)
Other errors → collect, continue + end + end + + opt deleteNamespace == true + CL ->> K8s: Delete Namespace + end + + CL -->> CO: CleanupResult{counts, errors, runIDs, deletedNames} + CO ->> Aud: LinkCleanupToRuns(execID, runIDs) + Aud ->> DB: UPDATE audit_log SET linked_run_ids = JSON array + CO ->> Aud: MarkVMsDeletedByName(vmNames, namespace, runIDs) + Aud ->> DB: UPDATE vm_details SET status='deleted', deleted_at + CO ->> Aud: MarkResourcesDeletedByName(type, names, namespace, runIDs) + Aud ->> DB: UPDATE resource_details SET status='deleted', deleted_at + CO ->> Aud: RecordCleanupCounts(execID, vms, svcs, secrets, dvs, pvcs, nsDeleted) + Aud ->> DB: UPDATE audit_log SET cleanup counters + CO ->> Aud: RecordEvent(execID, "cleanup_completed") + CO -->> CLI: CleanupResult + + %% ─── ERROR HANDLING PATTERNS ──────────────────────────────── + + Note over CLI, K8s: Error handling patterns for contributors + Note over VM: K8s API error classification:
─ AlreadyExists → success (idempotent create)
─ TooManyRequests / 5xx → retry with backoff
─ NotFound → fatal (on create) / skip (on delete)
─ Unauthorized / Forbidden → fatal, no retry + Note over RO: Concurrency: errgroup collects first error,
cancels context on failure, all goroutines respect ctx.Done() + Note over CL: Cleanup: continue-on-error pattern,
collect all errors, report at end + Note over Aud: Audit errors: logged via logger.Warn(),
never fail the operation — audit is observability, not control flow diff --git a/docs/mermaid/03-orchestrator-run-phases.mermaid b/docs/mermaid/03-orchestrator-run-phases.mermaid new file mode 100644 index 0000000..95ccfff --- /dev/null +++ b/docs/mermaid/03-orchestrator-run-phases.mermaid @@ -0,0 +1,191 @@ +%% Orchestrator Run Phases: the 5-phase pipeline inside RunOrchestrator.Run() +%% Shows concurrency boundaries, error handling, and data flow between phases + +sequenceDiagram + autonumber + + participant CLI as runE + participant RO as RunOrchestrator + participant Plan as planVMs + participant Reg as Registry + participant WL as Workload + participant NS as EnsureNamespace + participant Svc as CreateService + participant Sec as CreateCloudInitSecret + participant VM as CreateVM + participant Wait as WaitForAllVMsReady + participant Aud as Auditor + + CLI ->> RO: Run(ctx, execID, runID, workloadNames, vmCountFlag) + + %% ─── PHASE 1: PLAN ────────────────────────────────────────── + + Note over RO, WL: Phase 1 — planVMs (sequential, pure data) + RO ->> Plan: planVMs(ctx, execID, runID, names, vmCount, registry, opts) + + Plan ->> Reg: DefaultRegistry() + Note right of Reg: map[string]WorkloadFactory with 9 workload types + + loop For each workload name + Plan ->> Plan: Check config: workload disabled? + alt Disabled in config + Plan ->> Aud: RecordEvent("workload_skipped") + else Enabled + Plan ->> Plan: Merge global config with per-workload overrides + Plan ->> Reg: Get(name, mergedConfig, opts...) + Reg -->> Plan: Workload instance + + Plan ->> Aud: RecordWorkload(execID, WorkloadRecord) + Aud -->> Plan: workloadID + + alt Implements MultiVMWorkload + Plan ->> WL: RoleDistribution() + WL -->> Plan: [{role:"server", count:N}, {role:"client", count:N}] + loop For each role + Plan ->> WL: UserdataForRole(role, namespace) + WL -->> Plan: cloud-init YAML + loop For each VM in role count (0-indexed) + Plan ->> Plan: Build VMPlan
name: virtwork-{wl}-{role}-{i}
labels: managed-by, component, run-id, role + end + end + else Single-VM Workload + Plan ->> WL: CloudInitUserdata() + WL -->> Plan: cloud-init YAML + loop For each VM (0..VMCount-1) + Plan ->> WL: DataVolumeTemplates() + WL -->> Plan: []DataVolumeTemplateSpec (or nil) + Plan ->> Plan: Build VMPlan
name: virtwork-{wl}-{i}
NamespaceDataVolumes() uniquifies DVT names + end + end + end + end + + Plan -->> RO: []VMPlan, []vmNames,
map[name]Workload, map[name]workloadID + + %% ─── DRY RUN CHECK ────────────────────────────────────────── + + alt cfg.DryRun == true + RO ->> RO: printDryRun(plans) — marshal each VM spec to YAML + RO -->> CLI: RunResult (dry-run, no cluster changes) + end + + %% ─── PHASE 2: RESOURCES ───────────────────────────────────── + + Note over RO, Svc: Phase 2 — createResources (sequential) + RO ->> NS: EnsureNamespace(ctx, client, namespace, labels) + Note right of NS: Create or merge labels if exists
(AlreadyExists → patch) + NS -->> RO: OK + + loop For each workload where RequiresService() == true + RO ->> WL: ServiceSpec() + WL -->> RO: *corev1.Service + RO ->> RO: Inject run-id label into service + RO ->> Svc: CreateService(ctx, client, svc) + Note right of Svc: AlreadyExists → success (idempotent) + Svc -->> RO: OK + RO ->> Aud: RecordResource(execID, {type:"Service", name, ns}) + RO ->> Aud: RecordEvent(execID, "service_created") + end + + Note over RO: serviceCount accumulated + + %% ─── PHASE 3: SECRETS ─────────────────────────────────────── + + Note over RO, Sec: Phase 3 — createSecrets (parallel, errgroup limit=10) + Note right of RO: g, ctx := errgroup.WithContext(ctx)
g.SetLimit(10)
secretsCreated atomic.Int32 + + par Up to 10 concurrent goroutines + loop For each VMPlan[i] + RO ->> Sec: CreateCloudInitSecret(ctx, client,
"{vmName}-cloudinit", ns, userdata, labels) + Note right of Sec: Validates size ≤ 1 MiB
StringData["userdata"] = cloud-init YAML
AlreadyExists → success + Sec -->> RO: OK + RO ->> RO: plans[i].VMSpec.CloudInitSecretName = secretName + RO ->> Aud: RecordResource(execID, {type:"Secret", name, ns}) + RO ->> RO: secretsCreated.Add(1) + end + end + + RO ->> RO: g.Wait() — returns first error (cancels ctx) + + %% ─── PHASE 4: VMS ─────────────────────────────────────────── + + Note over RO, VM: Phase 4 — createVMs (parallel, errgroup limit=VMConcurrency) + Note right of RO: g, ctx := errgroup.WithContext(ctx)
g.SetLimit(cfg.VMConcurrency) + + par Up to VMConcurrency concurrent goroutines + loop For each VMPlan + RO ->> VM: BuildVMSpec(opts) → *kubevirtv1.VirtualMachine + RO ->> VM: CreateVM(ctx, client, vmObj) + Note right of VM: retryOnTransient: 1 + 5 retries
backoff: 1s, 2s, 4s, 8s, 16s
retries: TooManyRequests, ServerTimeout,
ServiceUnavailable, InternalError
AlreadyExists → success + + alt Success + RO ->> Aud: RecordVM(execID, wlID, VMRecord) + RO ->> Aud: RecordEvent(execID, "vm_created") + else Failure + RO ->> Aud: RecordEvent(execID, "vm_failed", error) + Note right of RO: Returns error → errgroup cancels ctx + end + end + end + + RO ->> RO: g.Wait() — returns first error + + alt createVMs failed + RO ->> RO: Loop all workload IDs + RO ->> Aud: UpdateWorkloadStatus(wlID, "failed") for each + RO -->> CLI: Return error + end + + %% ─── PHASE 5: READINESS ───────────────────────────────────── + + Note over RO, Wait: Phase 5 — waitForReadiness (WaitGroup + goroutine per VM) + + alt cfg.WaitForReady == true + RO ->> Wait: WaitForAllVMsReady(ctx, client, logger,
vmNames, ns, timeout, pollInterval) + Note right of Wait: sync.WaitGroup + sync.Mutex
One goroutine per VM name
Each defers recover() for panic safety + + par Independent goroutine per VM + loop Poll until Running or deadline + Wait ->> Wait: Get VMI by ObjectKey + alt VMI.Status.Phase == Running + Wait -->> Wait: nil (ready) + else NotFound + Wait ->> Wait: Sleep(interval), retry + else Other phase + Wait ->> Wait: Sleep(interval), retry + else Past deadline + Wait -->> Wait: ErrVMTimeout + end + end + end + + Wait -->> RO: map[vmName]error + + loop For each result + alt error == nil (VM ready) + RO ->> Aud: RecordEvent(execID, "vm_ready") + else error != nil (timeout/failure) + RO ->> Aud: RecordEvent(execID, "vm_timeout") + end + end + + alt Any failures + loop For each workload + alt Has failed VMs + RO ->> Aud: UpdateWorkloadStatus(wlID, "failed") + else All VMs ready + RO ->> Aud: UpdateWorkloadStatus(wlID, "created") + end + end + RO -->> RO: return ErrReadinessCheck (N of M VMs failed) + end + else --no-wait flag set + Note over RO: Skip readiness polling + end + + %% ─── COMPLETION ───────────────────────────────────────────── + + Note over RO, CLI: Completion + RO -->> CLI: RunResult{RunID, VMCount, ServiceCount, SecretCount} + CLI ->> Aud: CompleteExecution(execID, "success", "") + CLI ->> CLI: printSummary(result) diff --git a/docs/mermaid/04-config-resolution.mermaid b/docs/mermaid/04-config-resolution.mermaid new file mode 100644 index 0000000..82dee7f --- /dev/null +++ b/docs/mermaid/04-config-resolution.mermaid @@ -0,0 +1,106 @@ +%% Config Resolution: priority chain from CLI flags to final Config struct +%% Shows flag binding, env var mapping, YAML file parsing, and SSH key resolution + +sequenceDiagram + autonumber + + actor User + participant Cobra as Cobra Command + participant Viper as Viper Instance + participant Env as Environment + participant File as config.yaml + participant Defaults as SetDefaults + participant SSH as resolveSSHKeys + participant Config as Config struct + + %% ─── PRIORITY CHAIN (highest → lowest) ────────────────────── + + Note over User, Defaults: Priority chain: CLI flags > env vars > config file > defaults + + User ->> Cobra: virtwork run --workloads cpu,network
--cpu-cores 4 --memory 8Gi --namespace prod + + Note over Cobra: BindPersistentFlags registers:
--namespace, --kubeconfig, --config,
--verbose, --audit, --no-audit, --audit-db + + Note over Cobra: BindRunFlags registers:
--workloads, --vm-count, --cpu-cores,
--memory, --disk-size, --container-disk-image,
--dry-run, --no-wait, --timeout,
--ssh-user, --ssh-password, --ssh-key,
--ssh-key-file, --vm-concurrency + + Cobra ->> Viper: LoadConfig(cmd) begins + + %% ─── STEP 1: DEFAULTS ─────────────────────────────────────── + + Note over Viper, Defaults: Step 1 — Register defaults (lowest priority) + Viper ->> Defaults: SetDefaults(v) + Note right of Defaults: namespace → "virtwork"
container-disk-image → "quay.io/containerdisks/fedora:41"
disk-size → "10Gi"
cpu-cores → 2
memory → "2Gi"
wait-for-ready → true
timeout → 600
ssh-user → "virtwork"
audit → true
audit-db → "virtwork.db"
vm-concurrency → 10 + Defaults -->> Viper: Defaults registered + + %% ─── STEP 2: ENV VARS ─────────────────────────────────────── + + Note over Viper, Env: Step 2 — Bind environment variables + Viper ->> Viper: SetEnvPrefix("VIRTWORK") + Viper ->> Viper: SetEnvKeyReplacer("-" → "_") + Viper ->> Viper: AutomaticEnv() + Note right of Env: VIRTWORK_NAMESPACE=staging
VIRTWORK_CPU_CORES=2
VIRTWORK_SSH_AUTHORIZED_KEYS=key1,key2
VIRTWORK_KUBECONFIG=/path/to/kubeconfig + + %% ─── STEP 3: CONFIG FILE ──────────────────────────────────── + + Note over Viper, File: Step 3 — Read YAML config file (if --config provided) + alt --config flag set + Viper ->> File: ReadInConfig() + Note right of File: namespace: production
container-disk-image: registry.io/fedora:latest
cpu-cores: 2
memory: "4Gi"
workloads:
cpu:
enabled: true
vm_count: 3
cpu_cores: 8
memory: "16Gi"
network:
enabled: true
vm_count: 2
disk:
enabled: false
ssh-authorized-keys:
- "ssh-ed25519 AAAA... user@host" + File -->> Viper: Config merged (below env, above defaults) + else No --config flag + Note over Viper: Skip file — use env + defaults only + end + + %% ─── STEP 4: CLI FLAGS (HIGHEST PRIORITY) ─────────────────── + + Note over Cobra, Viper: Step 4 — Bind CLI flags (highest priority, only if Changed) + + Cobra ->> Viper: bindFlagIfSet for string flags + Note right of Cobra: namespace, kubeconfig,
container-disk-image, disk-size,
memory, ssh-user, ssh-password
Only binds if flag was explicitly
passed on CLI (cmd.Flag.Changed == true)
Prevents zero-value flags from
overriding env/file values + + Cobra ->> Viper: Check special flags: + Note right of Cobra: --cpu-cores: if Changed → v.Set
--timeout: if Changed → v.Set
--dry-run: if Changed → v.Set
--verbose: if Changed → v.Set
--no-wait: if Changed → WaitForReady=!val
--vm-concurrency: if Changed → v.Set + + %% ─── STEP 5: BUILD CONFIG STRUCT ──────────────────────────── + + Note over Viper, Config: Step 5 — Assemble Config struct from resolved values + Viper ->> Config: Build Config{} + Note right of Config: Namespace: resolved value
ContainerDiskImage: resolved value
DataDiskSize: resolved value
CPUCores: resolved value
Memory: resolved value
KubeconfigPath: resolved value
SSHUser, SSHPassword
WaitForReady, ReadyTimeoutSeconds
DryRun, Verbose
AuditEnabled, AuditDBPath
VMConcurrency + + %% ─── STEP 6: SSH KEY RESOLUTION ───────────────────────────── + + Note over Cobra, SSH: Step 6 — resolveSSHKeys (own priority chain) + Cobra ->> SSH: resolveSSHKeys(v, cmd) + + alt --ssh-key or --ssh-key-file flags set (CLI) + SSH ->> SSH: Collect inline keys from --ssh-key + SSH ->> SSH: Read files from --ssh-key-file, trim whitespace + SSH -->> Cobra: CLI keys (highest priority) + else VIRTWORK_SSH_AUTHORIZED_KEYS env var set + SSH ->> Env: Read env var, split on comma + SSH -->> Cobra: Env keys (medium priority) + else ssh-authorized-keys in YAML config + SSH ->> Viper: GetStringSlice("ssh-authorized-keys") + SSH -->> Cobra: File keys (lowest priority) + else None set + SSH -->> Cobra: Empty slice (SSH keys optional) + end + + Note over Config: cfg.SSHAuthorizedKeys = resolved keys + + %% ─── STEP 7: WORKLOADS MAP ────────────────────────────────── + + Viper ->> Config: Unmarshal workloads map from config file + Note right of Config: Workloads: map[string]WorkloadConfig
Each entry has: Enabled, VMCount,
CPUCores, Memory, Params + + %% ─── STEP 8: VALIDATION ───────────────────────────────────── + + Note over Config: Step 8 — Validate final Config + Config ->> Config: Validate() + Note right of Config: Namespace must not be empty
CPUCores must be ≥ 1
Memory must parse as k8s Quantity
DataDiskSize must parse as k8s Quantity
If WaitForReady: timeout ≥ 1s + Config -->> Cobra: *Config (validated) + + %% ─── PER-WORKLOAD CONFIG MERGE ────────────────────────────── + + Note over Config: Per-workload config merge (during planVMs) + Note right of Config: Global config provides defaults:
CPUCores=2, Memory="2Gi"

Per-workload overrides from YAML:
cpu: {cpu_cores: 8, memory: "16Gi"}

Merged WorkloadConfig sent to factory:
CPUCores=8, Memory="16Gi"

If per-workload field is zero/empty,
the global value fills in diff --git a/docs/mermaid/05-vm-lifecycle.mermaid b/docs/mermaid/05-vm-lifecycle.mermaid new file mode 100644 index 0000000..3e4120f --- /dev/null +++ b/docs/mermaid/05-vm-lifecycle.mermaid @@ -0,0 +1,152 @@ +%% VM Lifecycle: from spec construction through creation, readiness, and deletion +%% Shows retry logic, KubeVirt resource structure, and readiness polling + +sequenceDiagram + autonumber + + participant Orch as Orchestrator + participant Build as vm.BuildVMSpec + participant Crt as vm.CreateVM + participant Retry as retryOnTransient + participant K8s as K8s API Server + participant KV as KubeVirt Controller + participant VMI as VirtualMachineInstance + participant Poll as wait.WaitForVMReady + participant CL as cleanup.CleanupAll + + %% ─── VM SPEC CONSTRUCTION ─────────────────────────────────── + + Note over Orch, Build: VM Spec Construction + Orch ->> Build: BuildVMSpec(VMSpecOpts) + Note right of Build: Input: Name, Namespace, ContainerDiskImage,
CloudInitSecretName, CPUCores, Memory,
Labels, ExtraDisks, ExtraVolumes,
DataVolumeTemplates + + Build ->> Build: Validate inputs + Note right of Build: Name non-empty
Namespace non-empty
CPUCores > 0
ContainerDiskImage non-empty + + Build ->> Build: Construct base disks + Note right of Build: Disk 1: "containerdisk" (virtio bus)
Disk 2: "cloudinitdisk" (virtio bus)
+ any ExtraDisks from workload + + Build ->> Build: Construct volumes + Note right of Build: Vol 1: ContainerDisk{Image}
Vol 2: CloudInitNoCloud{SecretRef}
(or inline UserData if no secret)
+ any ExtraVolumes from workload + + Build ->> Build: Parse Memory as k8s Quantity + + Build ->> Build: Assemble VirtualMachine + Note right of Build: ObjectMeta: {Name, Namespace, Labels}
Spec:
RunStrategy: Always
Template.Spec.Domain:
CPU: {Cores: N}
Resources: {Requests: {memory: Quantity}}
Devices:
Disks: [container, cloudinit, extras...]
Interfaces: [{Name:"default", Masquerade}]
Networks: [{Name:"default", Pod}]
Template.Spec.Volumes: [above]
DataVolumeTemplates: [from workload] + + Build -->> Orch: *kubevirtv1.VirtualMachine + + %% ─── VM CREATION WITH RETRY ───────────────────────────────── + + Note over Orch, K8s: VM Creation (with transient error retry) + Orch ->> Crt: CreateVM(ctx, client, vmObj) + + Crt ->> Retry: retryOnTransient(ctx, createFn, maxRetries=5) + + loop Attempt 0..5 (1 + 5 retries) + Retry ->> Retry: Check ctx.Err() (cancelled?) + + Retry ->> Retry: Clear ResourceVersion (fresh create each attempt) + + Retry ->> K8s: client.Create(ctx, vm) + + alt 201 Created + K8s -->> Retry: Success + Retry -->> Crt: nil + else AlreadyExists + K8s -->> Retry: AlreadyExists error + Retry -->> Crt: nil (idempotent — treated as success) + else TooManyRequests / ServerTimeout / ServiceUnavailable / InternalError + K8s -->> Retry: Transient error + Note right of Retry: isTransientError() == true + + Retry ->> Retry: Calculate backoff
1s × 2^attempt
(1s, 2s, 4s, 8s, 16s) + + Retry ->> Retry: select {
case <-time.After(backoff):
case <-ctx.Done():
} + Note right of Retry: Respects context cancellation
during backoff sleep + else NotFound / Unauthorized / Forbidden / other permanent + K8s -->> Retry: Permanent error + Note right of Retry: isTransientError() == false + Retry -->> Crt: error (no retry) + end + end + + alt All retries exhausted + Retry -->> Crt: "max retries exceeded" wrapping last error + end + + Crt -->> Orch: nil or error + + %% ─── KUBEVIRT TAKES OVER ──────────────────────────────────── + + Note over K8s, VMI: KubeVirt controller reconciliation (cluster-side) + K8s ->> KV: VirtualMachine created event + KV ->> KV: Reconcile: RunStrategy=Always → create VMI + KV ->> VMI: Create VirtualMachineInstance + VMI ->> VMI: Phase: Pending → Scheduling → Scheduled + VMI ->> VMI: virt-launcher pod starts + VMI ->> VMI: Container disk pulled + VMI ->> VMI: Cloud-init userdata applied + VMI ->> VMI: DataVolumes provisioned (if any) + VMI ->> VMI: Phase: Running + Note right of VMI: Guest OS boots
Cloud-init installs packages
Systemd units start workload + + %% ─── READINESS POLLING ────────────────────────────────────── + + Note over Orch, VMI: Readiness Polling (per-VM goroutine) + Orch ->> Poll: WaitForVMReady(ctx, client, logger,
vmName, namespace, timeout, interval) + + Poll ->> Poll: deadline = now + timeout + + loop Until Running or deadline exceeded + Poll ->> Poll: Check ctx.Err() + + alt Past deadline + Poll -->> Orch: ErrVMTimeout + end + + Poll ->> K8s: Get VirtualMachineInstance(name, namespace) + + alt NotFound + Note right of Poll: VMI not yet created by controller + Poll ->> Poll: Log debug, sleep(interval) + else Get error + Poll -->> Orch: error + else VMI.Status.Phase == Running + Poll -->> Orch: nil (VM is ready) + else Other phase (Pending, Scheduling, Scheduled) + Note right of Poll: VM still booting + Poll ->> Poll: sleep(interval) via select:
case <-time.After(interval):
case <-ctx.Done(): + end + end + + %% ─── VM DELETION (CLEANUP) ────────────────────────────────── + + Note over Orch, K8s: VM Deletion (during cleanup) + Orch ->> CL: CleanupAll(ctx, client, config, deleteNS, runID) + CL ->> K8s: List VMs by label selector + K8s -->> CL: vmList + + loop For each VM in vmList + CL ->> K8s: client.Delete(ctx, &vm) + alt Success + K8s -->> CL: OK + else NotFound + K8s -->> CL: NotFound → skip (idempotent) + else Other error + CL ->> CL: Collect error, continue + end + end + + CL -->> Orch: CleanupResult (counts, errors, runIDs) + + Note over K8s, VMI: KubeVirt controller reconciliation + K8s ->> KV: VirtualMachine deleted + KV ->> VMI: Delete VirtualMachineInstance + VMI ->> VMI: virt-launcher pod terminated + VMI ->> VMI: Phase: (removed) + + %% ─── DATA VOLUME TEMPLATE ─────────────────────────────────── + + Note over Build: DataVolumeTemplate construction (for disk/database workloads) + Note right of Build: BuildDataVolumeTemplate(name, size):
Parse size as k8s Quantity
Return DataVolumeTemplateSpec{
Name: name,
Source: {Blank: {}},
Storage: {Resources: {Requests: size}}
}

CDI controller provisions the PVC
automatically when VM is created diff --git a/docs/mermaid/06-audit-lifecycle.mermaid b/docs/mermaid/06-audit-lifecycle.mermaid new file mode 100644 index 0000000..fbf6fe6 --- /dev/null +++ b/docs/mermaid/06-audit-lifecycle.mermaid @@ -0,0 +1,183 @@ +%% Audit Lifecycle: schema migration, recording touchpoints during run and cleanup +%% Shows the full audit trail from database init through execution completion + +sequenceDiagram + autonumber + + participant CLI as cmd/virtwork + participant Init as initAuditor + participant SA as SQLiteAuditor + participant Mig as migrateDB + participant DB as SQLite DB + participant RO as RunOrchestrator + participant CO as CleanupOrchestrator + + %% ─── DATABASE INITIALIZATION ──────────────────────────────── + + Note over CLI, DB: Database initialization and schema migration + CLI ->> Init: initAuditor(cmd, cfg) + + alt --no-audit flag or cfg.AuditEnabled == false + Init -->> CLI: NoOpAuditor{} (all methods are no-ops) + else Audit enabled + Init ->> SA: NewSQLiteAuditor(dbPath) + SA ->> SA: MkdirAll for DB directory (0o750) + SA ->> DB: sql.Open("sqlite", DSN) + Note right of DB: DSN: path?_pragma=journal_mode(WAL)
&_pragma=foreign_keys(on) + + SA ->> Mig: migrateDB(db) + Mig ->> DB: CREATE TABLE IF NOT EXISTS schema_version + Mig ->> DB: SELECT COUNT(*) FROM schema_version + + alt No rows yet + Mig ->> DB: INSERT INTO schema_version (version) VALUES (0) + end + + Mig ->> DB: SELECT version FROM schema_version + DB -->> Mig: version = N + + loop For each unapplied migration (version N → latest) + Mig ->> DB: BEGIN TRANSACTION + Mig ->> DB: Execute migration function + Note right of DB: migrateV1Baseline executes SchemaSQL:
─ audit_log (execution records)
─ workload_details (per-workload config)
─ vm_details (individual VMs)
─ resource_details (services, secrets)
─ events (fine-grained event log)
─ Indexes on started_at, namespace,
status, run_id, event_type, etc. + Mig ->> DB: UPDATE schema_version SET version = N+1 + Mig ->> DB: COMMIT + end + + Mig -->> SA: Migration complete + SA -->> Init: *SQLiteAuditor ready + Init -->> CLI: Auditor + end + + %% ─── RUN EXECUTION AUDIT TRAIL ────────────────────────────── + + Note over CLI, DB: Audit trail during "virtwork run" + + CLI ->> SA: StartExecution(ctx, "run", cfg) + SA ->> SA: Generate runID = uuid.New() + SA ->> SA: Extract workloads CSV, SSH configured flag + SA ->> DB: INSERT INTO audit_log
(run_id, command, status='in_progress',
kubeconfig_path, cluster_context, namespace,
container_disk_image, default_cpu_cores,
default_memory, data_disk_size, workloads_csv,
dry_run, ssh_auth_configured, cleanup_mode,
wait_for_ready, ready_timeout_seconds, started_at) + DB -->> SA: executionID (auto-increment) + SA -->> CLI: (executionID, runID) + + Note over RO: Orchestrator records as it works + + loop For each workload (planVMs phase) + RO ->> SA: RecordWorkload(execID, WorkloadRecord) + SA ->> DB: INSERT INTO workload_details
(audit_id, workload_type, enabled, vm_count,
cpu_cores, memory, has_data_disk, data_disk_size,
requires_service, status='pending') + DB -->> SA: workloadID + end + + opt Workload skipped (disabled in config) + RO ->> SA: RecordEvent(execID, {type:"workload_skipped"}) + SA ->> DB: INSERT INTO events
(audit_id, vm_id, workload_id, event_type,
message, error_detail, occurred_at) + end + + RO ->> SA: RecordEvent(execID, {type:"execution_started"}) + + loop For each service created + RO ->> SA: RecordResource(execID, {type:"Service", name, ns}) + SA ->> DB: INSERT INTO resource_details
(audit_id, resource_type, resource_name,
namespace, status='created', created_at) + DB -->> SA: resourceID + + RO ->> SA: RecordEvent(execID, {type:"service_created"}) + end + + loop For each cloud-init secret created + RO ->> SA: RecordResource(execID, {type:"Secret", name, ns}) + SA ->> DB: INSERT INTO resource_details (same schema) + end + + loop For each VM created + RO ->> SA: RecordVM(execID, wlID, VMRecord) + SA ->> DB: INSERT INTO vm_details
(audit_id, workload_id, vm_name, namespace,
component, role, cpu_cores, memory,
container_disk_image, has_data_disk,
data_disk_size, status='created', created_at) + DB -->> SA: vmID + + RO ->> SA: RecordEvent(execID, {type:"vm_created"}) + SA ->> DB: INSERT INTO events
(audit_id, vm_id, event_type, occurred_at) + end + + alt VM creation fails + RO ->> SA: RecordEvent(execID, {type:"vm_failed", error}) + loop For each workload + RO ->> SA: UpdateWorkloadStatus(wlID, "failed") + SA ->> DB: UPDATE workload_details SET status='failed' + end + end + + opt Readiness polling enabled + loop For each VM result + alt VM ready + RO ->> SA: RecordEvent(execID, {type:"vm_ready"}) + else VM timed out + RO ->> SA: RecordEvent(execID, {type:"vm_timeout"}) + end + end + + alt Any failures + loop For each workload + alt Has failed VMs + RO ->> SA: UpdateWorkloadStatus(wlID, "failed") + else All VMs ready + RO ->> SA: UpdateWorkloadStatus(wlID, "created") + end + end + end + end + + alt No failures (all VMs created and ready) + loop For each workload + RO ->> SA: UpdateWorkloadStatus(wlID, "created") + SA ->> DB: UPDATE workload_details SET status='created' + end + end + + CLI ->> SA: CompleteExecution(execID, "success", "") + SA ->> DB: UPDATE audit_log
SET status='success', completed_at=now() + + %% ─── CLEANUP EXECUTION AUDIT TRAIL ────────────────────────── + + Note over CLI, DB: Audit trail during "virtwork cleanup" + + CLI ->> SA: StartExecution(ctx, "cleanup", cfg) + SA ->> DB: INSERT INTO audit_log
(run_id, command='cleanup', status='in_progress',
full column set same as run, started_at) + DB -->> SA: executionID + + CO ->> SA: RecordEvent(execID, {type:"cleanup_started",
message: "namespace=X, run-id=Y"}) + + Note over CO: Cleanup executes, collects run IDs from resources + + CO ->> SA: LinkCleanupToRuns(execID, ["uuid-1", "uuid-2"]) + SA ->> DB: UPDATE audit_log
SET linked_run_ids = '["uuid-1","uuid-2"]' + Note right of DB: linked_run_ids is JSON TEXT
(PostgreSQL JSONB compatible) + + CO ->> SA: MarkVMsDeletedByName(vmNames, namespace, runIDs) + SA ->> DB: UPDATE vm_details SET status='deleted', deleted_at
WHERE vm_name IN (...) AND namespace=?
AND audit_id IN (SELECT id FROM audit_log WHERE run_id IN (...)) + + CO ->> SA: MarkResourcesDeletedByName(type, names, namespace, runIDs) + SA ->> DB: UPDATE resource_details SET status='deleted', deleted_at
WHERE resource_name IN (...) AND resource_type=?
AND audit_id IN (SELECT id FROM audit_log WHERE run_id IN (...)) + Note right of DB: Called per resource type:
Service, Secret, DataVolume, PersistentVolumeClaim + + CO ->> SA: RecordCleanupCounts(execID, vms, svcs, secrets, dvs, pvcs, nsDeleted) + SA ->> DB: UPDATE audit_log
SET vms_deleted=N, services_deleted=N,
secrets_deleted=N, dvs_deleted=N,
pvcs_deleted=N, namespace_deleted=0|1 + + CO ->> SA: RecordEvent(execID, {type:"cleanup_completed"}) + + CLI ->> SA: CompleteExecution(execID, "success", "") + SA ->> DB: UPDATE audit_log
SET status='success', completed_at=now() + + %% ─── ERROR COMPLETION ─────────────────────────────────────── + + Note over CLI, DB: Error completion (either command) + alt Execution fails at any point + CLI ->> SA: CompleteExecution(execID, "error", errSummary) + SA ->> DB: UPDATE audit_log
SET status='error', completed_at=now(),
error_summary='...' + end + + CLI ->> SA: Close() + SA ->> DB: Close database connection + + %% ─── AUDIT ERROR HANDLING ─────────────────────────────────── + + Note over RO, SA: Audit error handling policy + Note right of SA: Audit is observability, not control flow.
All audit errors are logged via logger.Warn()
and never block the main operation.

If the auditor fails, the run/cleanup
continues — data is best-effort recorded. diff --git a/docs/mermaid/07-cloudinit-generation.mermaid b/docs/mermaid/07-cloudinit-generation.mermaid new file mode 100644 index 0000000..8165200 --- /dev/null +++ b/docs/mermaid/07-cloudinit-generation.mermaid @@ -0,0 +1,124 @@ +%% Cloud-Init Generation: how cloud-init YAML is assembled per workload type +%% Shows the BuildCloudConfig pipeline, SSH injection, and workload-specific content + +sequenceDiagram + autonumber + + participant Orch as Orchestrator + participant WL as Workload + participant Base as BaseWorkload + participant CI as cloudinit.BuildCloudConfig + participant YAML as YAML Marshal + + %% ─── SINGLE-VM WORKLOAD (CPU example) ─────────────────────── + + Note over Orch, YAML: Single-VM workload cloud-init (e.g. CPUWorkload) + + Orch ->> WL: CloudInitUserdata() + + WL ->> Base: BuildCloudConfig(CloudConfigOpts) + Note right of WL: CPUWorkload provides:
Packages: ["stress-ng"]
WriteFiles: [{
Path: "/etc/systemd/system/virtwork-cpu.service",
Content: [Unit]...[Service]
ExecStart=/usr/bin/stress-ng --cpu 0
--cpu-method all --timeout 0
Permissions: "0644"
}]
RunCmd: [
["systemctl","daemon-reload"],
["systemctl","enable","--now","virtwork-cpu.service"]
] + + Base ->> Base: Inject SSH credentials into opts + Note right of Base: If SSHUser set:
opts.SSHUser = "virtwork"
opts.SSHPassword = (if configured)
opts.SSHAuthorizedKeys = [keys...] + + Base ->> CI: BuildCloudConfig(opts) + + %% ─── CLOUD-INIT BUILD PIPELINE ────────────────────────────── + + Note over CI, YAML: BuildCloudConfig assembly pipeline + + CI ->> CI: Initialize empty doc map + + CI ->> CI: Add packages (if non-empty) + Note right of CI: packages:
- stress-ng + + CI ->> CI: Add write_files (if non-empty) + Note right of CI: write_files:
- path: /etc/systemd/system/virtwork-cpu.service
content: |
[Unit]
Description=virtwork CPU workload
...
permissions: "0644" + + CI ->> CI: Add runcmd (if non-empty) + Note right of CI: runcmd:
- [systemctl, daemon-reload]
- [systemctl, enable, --now, virtwork-cpu.service] + + CI ->> CI: Build SSH user configuration + alt SSHUser is set + CI ->> CI: Create user object + Note right of CI: users:
- name: virtwork
sudo: "ALL=(ALL) NOPASSWD:ALL"
shell: /bin/bash + + alt SSHPassword is set + CI ->> CI: Enable password auth + Note right of CI: lock_passwd: false
plain_text_passwd: "password"
ssh_pwauth: true + else No password + CI ->> CI: Lock password + Note right of CI: lock_passwd: true + end + + alt SSHAuthorizedKeys is non-empty + CI ->> CI: Add authorized keys + Note right of CI: ssh_authorized_keys:
- "ssh-ed25519 AAAA... user@host" + end + end + + CI ->> CI: Validate Extra map + Note right of CI: Rejects reserved keys:
packages, write_files, runcmd,
users, ssh_pwauth
→ ErrReservedKey + + CI ->> CI: Merge Extra keys via maps.Copy() + + CI ->> YAML: yaml.Marshal(doc) + YAML -->> CI: YAML bytes + + CI -->> Base: "#cloud-config\n" + YAML string + Base -->> WL: cloud-init YAML + WL -->> Orch: cloud-init YAML + + %% ─── MULTI-VM WORKLOAD (Network example) ─────────────────── + + Note over Orch, YAML: Multi-VM workload cloud-init (e.g. NetworkWorkload) + + Orch ->> WL: UserdataForRole("server", namespace) + WL ->> Base: BuildCloudConfig(serverOpts) + Note right of WL: Server opts:
Packages: ["iperf3"]
WriteFiles: [systemd unit with
ExecStart=/usr/bin/iperf3 -s]
RunCmd: [enable service] + Base ->> CI: BuildCloudConfig(opts) + CI -->> Orch: server cloud-init YAML + + Orch ->> WL: UserdataForRole("client", namespace) + WL ->> Base: BuildCloudConfig(clientOpts) + Note right of WL: Client opts:
Packages: ["iperf3"]
WriteFiles: [systemd unit with
ExecStart=bash loop:
iperf3 -c
virtwork-iperf3-server.{ns}.svc.cluster.local
-t 60 -P 4 --bidir
+ restart loop with 10s sleep]
RunCmd: [enable service] + Base ->> CI: BuildCloudConfig(opts) + CI -->> Orch: client cloud-init YAML + + %% ─── DATA DISK WORKLOAD (Disk example) ────────────────────── + + Note over Orch, YAML: Data disk workload cloud-init (e.g. DiskWorkload) + + Orch ->> WL: CloudInitUserdata() + WL ->> Base: BuildCloudConfig(diskOpts) + Note right of WL: Disk opts:
Packages: ["fio"]
WriteFiles: [
fio mixed R/W profile,
fio sequential write profile,
systemd unit:
ExecStartPre=disk setup script
(mkfs.xfs + mount /data)
ExecStart=bash loop:
fio mixed-rw.fio; sleep 10;
fio seq-write.fio; sleep 10
]
RunCmd: [enable service] + Base ->> CI: BuildCloudConfig(opts) + CI -->> Orch: disk cloud-init YAML + + %% ─── DATABASE WORKLOAD ────────────────────────────────────── + + Note over Orch, YAML: Database workload cloud-init (e.g. DatabaseWorkload) + + Orch ->> WL: CloudInitUserdata() + WL ->> Base: BuildCloudConfig(dbOpts) + Note right of WL: Database opts:
Packages: ["postgresql-server"]
WriteFiles: [
db setup script (initdb, create pgbench db,
scale factor 50, start postgres),
systemd unit:
ExecStartPre=setup script
ExecStart=bash loop:
pgbench -c 10 -j 2 -T 300 pgbench
+ sleep 10 between cycles
]
RunCmd: [enable service] + Base ->> CI: BuildCloudConfig(opts) + CI -->> Orch: database cloud-init YAML + + %% ─── TPS WORKLOAD ─────────────────────────────────────────── + + Note over Orch, YAML: TPS workload cloud-init (e.g. TPSWorkload) + + Orch ->> WL: UserdataForRole("server", namespace) + WL ->> Base: BuildCloudConfig(tpsServerOpts) + Note right of WL: TPS Server opts:
Packages: ["netperf","python3"]
WriteFiles: [
server script:
netserver on port 12865
dd to generate test file
python3 -m http.server 8080
systemd unit:
ExecStart=/usr/local/bin/virtwork-tps-server.sh
] + + Orch ->> WL: UserdataForRole("client", namespace) + WL ->> Base: BuildCloudConfig(tpsClientOpts) + Note right of WL: TPS Client opts:
Packages: ["netperf","curl"]
WriteFiles: [
client script:
wait for server HTTP
netperf TCP_RR tests
HTTP file transfer tests
measure TPS + throughput
systemd unit:
ExecStart=/usr/local/bin/virtwork-tps-client.sh
Params from config:
file-size (default "10M")
iterations (default "30")
duration (default "60")
] + + %% ─── SECRET STORAGE ───────────────────────────────────────── + + Note over Orch: Cloud-init delivery to VM + Note right of Orch: Cloud-init YAML stored as K8s Secret:
Name: {vmName}-cloudinit
StringData["userdata"]: YAML content
Size limit: ≤ 1 MiB

VM references secret via:
CloudInitNoCloud.UserDataSecretRef

KubeVirt injects as NoCloud datasource
at VM boot time diff --git a/docs/mermaid/08-cleanup-mechanics.mermaid b/docs/mermaid/08-cleanup-mechanics.mermaid new file mode 100644 index 0000000..1eb26d9 --- /dev/null +++ b/docs/mermaid/08-cleanup-mechanics.mermaid @@ -0,0 +1,198 @@ +%% Cleanup Mechanics: label-based discovery, ordered deletion, and run-id linking +%% Shows the full cleanup flow including preview, confirmation, and audit integration + +sequenceDiagram + autonumber + + actor User + participant CLI as cleanupE + participant CO as CleanupOrchestrator + participant CL as cleanup pkg + participant K8s as K8s API Server + participant Aud as SQLiteAuditor + participant DB as SQLite DB + + %% ─── ENTRY AND MODE SELECTION ─────────────────────────────── + + Note over User, CLI: Entry point and cleanup mode determination + User ->> CLI: virtwork cleanup [flags] + + CLI ->> CLI: config.LoadConfig(cmd) + CLI ->> CLI: initAuditor(cmd, cfg) + CLI ->> CLI: cluster.Connect(kubeconfigPath) + + CLI ->> CLI: Determine cleanup mode + alt --dry-run flag set + Note right of CLI: mode = "dry-run"
Preview only, no deletions + else --run-id UUID provided + Note right of CLI: mode = "run-id"
Targeted cleanup of specific run
Skips confirmation prompt + else No --run-id + Note right of CLI: mode = "all"
Clean all managed resources
Requires confirmation prompt + end + + CLI ->> Aud: StartExecution(ctx, "cleanup", cfg) + Aud ->> DB: INSERT audit_log (command='cleanup', cleanup_mode) + DB -->> CLI: executionID + + %% ─── PREVIEW PHASE (READ-ONLY) ───────────────────────────── + + Note over CLI, K8s: Preview phase — discover resources without modifying + CLI ->> CO: Preview(ctx, execID, targetRunID) + CO ->> CL: PreviewCleanup(ctx, client, config, runID) + + CL ->> CL: Build label selector + Note right of CL: Base labels:
app.kubernetes.io/managed-by: virtwork

If runID provided, add:
virtwork/run-id: {UUID} + + CL ->> K8s: List VirtualMachines (namespace, labels) + K8s -->> CL: VM list + CL ->> K8s: List Services (namespace, labels) + K8s -->> CL: Service list + CL ->> K8s: List Secrets (namespace, labels) + K8s -->> CL: Secret list + CL ->> K8s: List DataVolumes (namespace, labels) + K8s -->> CL: DataVolume list + CL ->> K8s: List PersistentVolumeClaims (namespace, labels) + K8s -->> CL: PVC list + + loop For each resource across all types + CL ->> CL: collectRunID(resource.Labels, runIDSet) + Note right of CL: Extract virtwork/run-id label
Add to deduplication set + end + + CL -->> CO: CleanupPreview{
VMCount, ServiceCount, SecretCount,
DVCount, PVCCount, TotalCount,
RunIDs: [unique UUIDs]
} + CO -->> CLI: CleanupPreview + + %% ─── USER CONFIRMATION ────────────────────────────────────── + + Note over User, CLI: Confirmation gate + CLI ->> CLI: printCleanupPreview(preview) + Note right of CLI: Displays:
Resources to delete:
VMs: 6
Services: 2
Secrets: 6
DataVolumes: 2
PVCs: 2
Total: 18 + + alt TotalCount == 0 + CLI -->> User: "nothing to clean up" + CLI ->> Aud: CompleteExecution(execID, "success", "") + Note over CLI: Return early — nothing to delete + else DryRun mode + CLI -->> User: "dry-run mode — no resources were deleted" + CLI ->> Aud: CompleteExecution(execID, "success", "") + Note over CLI: Return early — preview only + else --run-id provided (targeted cleanup) + Note right of CLI: Skip confirmation — targeted mode
is considered intentional + else --yes flag provided + Note right of CLI: Skip confirmation — auto-approved + else All mode (no --run-id, no --yes) + CLI ->> User: "Proceed with deletion? (yes/NO):" + User ->> CLI: PromptForConfirmation() + alt User confirms + Note right of CLI: Proceed with deletion + else User declines + CLI -->> User: "cleanup aborted by user" + CLI ->> Aud: CompleteExecution(execID, "aborted", "user declined confirmation") + Note over CLI: Return early + end + end + + %% ─── EXECUTION PHASE (ORDERED DELETION) ───────────────────── + + Note over CLI, K8s: Execute phase — ordered resource deletion + CLI ->> CO: Execute(ctx, execID, deleteNamespace, targetRunID) + CO ->> Aud: RecordEvent(execID, "cleanup_started") + + CO ->> CL: CleanupAll(ctx, client, config, deleteNS, runID) + + CL ->> CL: Build label selector (same as preview) + + Note over CL, K8s: Deletion order is intentional:
1. VMs — stop workloads first
2. Services — remove networking
3. Secrets — remove cloud-init config
4. DataVolumes — release storage (before PVCs)
5. PVCs — release persistent volume claims
6. Namespace — final sweep (if requested) + + Note over CL, K8s: Step 1: Delete VirtualMachines + CL ->> K8s: List VMs (namespace, labels) + K8s -->> CL: VM list + loop For each VM + CL ->> CL: collectRunID(vm.Labels, runIDSet) + CL ->> K8s: Delete VM + alt Success + CL ->> CL: vmsDeleted++ + else NotFound + Note right of CL: Skip — already deleted (idempotent) + else Other error + CL ->> CL: errors = append(errors, err) + Note right of CL: Continue — don't abort on single failure + end + end + + Note over CL, K8s: Step 2: Delete Services + CL ->> K8s: List Services (namespace, labels) + loop For each Service + CL ->> CL: collectRunID + Delete + Note right of CL: Same error handling pattern + end + + Note over CL, K8s: Step 3: Delete Secrets + CL ->> K8s: List Secrets (namespace, labels) + loop For each Secret + CL ->> CL: collectRunID + Delete + end + + Note over CL, K8s: Step 4: Delete DataVolumes (before PVCs) + CL ->> K8s: List DataVolumes (namespace, labels) + Note right of CL: DVs deleted before PVCs because
CDI controller may own PVCs via
DataVolume → deleting DV first
lets controller GC its owned PVC + loop For each DataVolume + CL ->> CL: collectRunID + Delete + end + + Note over CL, K8s: Step 5: Delete PVCs + CL ->> K8s: List PVCs (namespace, labels) + loop For each PVC + CL ->> CL: collectRunID + Delete + end + + opt deleteNamespace == true + Note over CL, K8s: Step 6: Delete Namespace (final sweep) + CL ->> K8s: Delete Namespace + alt Success + CL ->> CL: namespaceDeleted = true + else NotFound + Note right of CL: Already deleted — OK + else Error + CL ->> CL: errors = append(errors, err) + end + end + + CL -->> CO: CleanupResult{
VMsDeleted, ServicesDeleted,
SecretsDeleted, DVsDeleted,
PVCsDeleted, NamespaceDeleted,
Errors: [...],
RunIDs: [unique UUIDs from all resources],
DeletedVMNames, DeletedServiceNames,
DeletedSecretNames, DeletedDVNames,
DeletedPVCNames
} + + %% ─── AUDIT RECORDING ──────────────────────────────────────── + + Note over CO, DB: Audit recording after cleanup + + alt result.RunIDs is non-empty + CO ->> Aud: LinkCleanupToRuns(execID, result.RunIDs) + Aud ->> DB: UPDATE audit_log
SET linked_run_ids = '["uuid-1","uuid-2",...]' + Note right of DB: JSON array stored as TEXT
Links this cleanup to the specific runs
whose resources were deleted + end + + CO ->> Aud: MarkVMsDeletedByName(vmNames, namespace, runIDs) + Aud ->> DB: UPDATE vm_details SET status='deleted', deleted_at
WHERE vm_name IN (...) AND namespace=?
AND audit_id IN (SELECT id FROM audit_log WHERE run_id IN (...)) + + loop For each resource type (Service, Secret, DataVolume, PVC) + CO ->> Aud: MarkResourcesDeletedByName(type, names, namespace, runIDs) + Aud ->> DB: UPDATE resource_details SET status='deleted', deleted_at
WHERE resource_name IN (...) AND resource_type=?
AND audit_id scoped by run_id + end + + CO ->> Aud: RecordCleanupCounts(execID,
vms, svcs, secrets, dvs, pvcs, nsDeleted) + Aud ->> DB: UPDATE audit_log
SET vms_deleted=N, services_deleted=N, ... + + CO ->> Aud: RecordEvent(execID, "cleanup_completed") + + CO -->> CLI: CleanupResult + CLI ->> Aud: CompleteExecution(execID, "success", "") + + alt Any errors in CleanupResult + CLI -->> User: Warnings: [individual deletion errors] + end + + CLI -->> User: Cleanup summary:
Deleted N VMs, M services, ... + + %% ─── RUN-ID TRACEABILITY ──────────────────────────────────── + + Note over User, DB: Run-ID traceability chain + Note right of DB: Every virtwork run generates a UUID (run-id)

run-id is applied as a K8s label to every
resource created during that run:
virtwork/run-id: "abc-123-..."

Cleanup collects run-ids from the
resources it deletes and stores them
in audit_log.linked_run_ids as JSON

This creates a chain:
run audit_log.run_id = "abc-123"
↓ labels on K8s resources
cleanup audit_log.linked_run_ids = ["abc-123"]

Query: which cleanup deleted run X?
SELECT * FROM audit_log
WHERE linked_run_ids LIKE '%abc-123%'