Skip to content

feat: cluster test framework #545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DaRacci wants to merge 12 commits into
base: master
Choose a base branch
from
Open

feat: cluster test framework #545

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
adcb184
feat: cluster test framework
DaRacci Mar 7, 2026
b01e734
chore: openspec for testing framework
DaRacci Apr 21, 2026
de92f15
docs(testing): update design docs for pre-deploy VM integration tests
DaRacci Jun 28, 2026
5516012
feat(vm-test): add NixOS VM integration testing framework for server …
DaRacci Jun 28, 2026
7171bea
refactor(test-vm): remove unused inputs param, humanize comments
DaRacci Jun 28, 2026
96be296
fix(test-vm): wire server.tests.units, fix postgres-backup scenario
DaRacci Jun 28, 2026
bfeb083
fix(test-vm): inject scenario name from directory
DaRacci Jun 28, 2026
f0978b2
docs(openspec): comprehensive-server-tests proposal, design, tasks
DaRacci Jun 28, 2026
011e1f6
feat(test): Phase 1+2 server tests — 56 targets, all pass QEMU
DaRacci Jun 28, 2026
7135633
refactor: make external imports conditional and streamline VM test in…
DaRacci Jun 29, 2026
6a41e02
test: overhaul VM tests with real sops-encrypted dummy secrets and im…
DaRacci Jun 30, 2026
d600b72
refactor(vm-tests): drop upstream-only test scenarios and auto-discov…
DaRacci Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .woodpecker/test-vm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# PR-only, KVM.
# Build nixosTestConfigurations.* for all server hosts.
# TODO: affected-host build (PR-changed only).
#
# Independent of check.yaml.
Comment on lines +1 to +5

@coderabbitai coderabbitai Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🔵 Trivial

TODO acknowledges full-host builds; currently every PR builds all server + scenario hosts.

Per the flake/nixos/flake-module.nix snippet, nixosTestConfigurations enumerates every server host plus every scenario directory, so this pipeline will grow linearly (VM builds are expensive) as hosts/scenarios are added. Want help sketching an affected-host filter (e.g., diffing hosts/server/** against the PR's changed files) to close the TODO?

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.woodpecker/test-vm.yaml around lines 1 - 5, The PR-only KVM job currently
builds every server and scenario host via nixosTestConfigurations, which does
not scale as the repo grows. Update the test-vm pipeline to run only affected
hosts by filtering against the PR’s changed files (for example, comparing
changes under hosts/server/** and scenario directories) before expanding the
build set. Keep the logic anchored around the test-vm job and the
nixosTestConfigurations consumer so the affected-host selection happens before
VM builds are scheduled.


labels:
platform: linux/amd64
kvm: true

when:
- event: pull_request

steps:
- name: Build VM tests
image: registry.racci.dev/lix-woodpecker:latest
pull: true
environment:
binary_cache_token:
from_secret: binary_cache_token
commands: |
nix run .#setup-attic -- --watch
nix run .#archive-flakes -- "." "flake/nixos"

- name: Run VM tests
image: registry.racci.dev/lix-woodpecker:latest
commands: |
for host in $(nix eval .#nixosTestConfigurations --apply 'builtins.attrNames' --json | jq -r '.[]'); do
echo "::group::VM test: $host"
nix build ".#nixosTestConfigurations.$host" --no-link -L || echo "FAILED: $host"
echo "::endgroup::"
done
Comment on lines +28 to +32

@coderabbitai coderabbitai Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🔵 Trivial

Sequential per-host loop won't scale as hosts/scenarios grow.

All builds run serially in a single shell loop on one agent, so total CI time grows linearly with the host count. Native Woodpecker matrix builds require statically-known values at parse time, which doesn't fit the dynamically discovered host list here (nix eval .#nixosTestConfigurations), so this isn't a drop-in fix — but worth considering a generated/dynamic pipeline step or splitting into parallel agent runs if VM build times become a bottleneck.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.woodpecker/test-vm.yaml around lines 28 - 32, The VM test loop in the test
step runs each host serially, so it will become a CI bottleneck as the host list
grows. Update the logic around the host iteration in the test script to use a
parallelized or dynamically generated execution strategy instead of a single
shell loop in one agent, while keeping the current `nix eval
.#nixosTestConfigurations` host discovery and `nix build` per-host behavior
intact.

🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Per-host build failures are swallowed — the step always reports success.

Woodpecker runs step commands as a single /bin/sh -e script, and a step's overall exit status is what determines pipeline pass/fail. Here, nix build ... || echo "FAILED: $host" suppresses the non-zero exit from -e, and the loop's last command (echo "::endgroup::") always returns 0. Consequently, this "Run VM tests" step will report success even if every host build fails, defeating the purpose of gating PRs on VM test results.

🐛 Proposed fix to propagate failures 
   - name: Run VM tests
     image: registry.racci.dev/lix-woodpecker:latest
     commands: |
+      status=0
       for host in $(nix eval .#nixosTestConfigurations --apply 'builtins.attrNames' --json | jq -r '.[]'); do
         echo "::group::VM test: $host"
-        nix build ".#nixosTestConfigurations.$host" --no-link -L || echo "FAILED: $host"
+        if ! nix build ".#nixosTestConfigurations.$host" --no-link -L; then
+          echo "FAILED: $host"
+          status=1
+        fi
         echo "::endgroup::"
       done
+      exit $status
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for host in $(nix eval .#nixosTestConfigurations --apply 'builtins.attrNames' --json | jq -r '.[]'); do
echo "::group::VM test: $host"
nix build ".#nixosTestConfigurations.$host" --no-link -L || echo "FAILED: $host"
echo "::endgroup::"
done
status=0
for host in $(nix eval .#nixosTestConfigurations --apply 'builtins.attrNames' --json | jq -r '.[]'); do
echo "::group::VM test: $host"
if ! nix build ".#nixosTestConfigurations.$host" --no-link -L; then
echo "FAILED: $host"
status=1
fi
echo "::endgroup::"
done
exit $status
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.woodpecker/test-vm.yaml around lines 28 - 32, The “Run VM tests” loop is
swallowing failures because the `nix build` error is followed by a successful
`echo`, so the shell step always exits 0. Update the VM test commands in
`.woodpecker/test-vm.yaml` to let `nix build` failures propagate from the loop
instead of masking them, and ensure the step exits non-zero when any host build
fails. Use the existing `nix build ".#nixosTestConfigurations.$host"` loop and
the `echo "::group::VM test: $host"` / `echo "::endgroup::"` structure as the
location to fix this behavior.

40 changes: 40 additions & 0 deletions _run_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# QEMU VM integration test runner - sequential
# Usage: bash _run_tests.sh

set -o pipefail

HOSTS=("nixai" "nixmon" "nixdev" "nixio")
Comment thread
DaRacci marked this conversation as resolved.
RESULTS=()

for host in "${HOSTS[@]}"; do
echo ""
echo "============================================"
echo "=== BUILDING: ${host} ==="
echo "============================================"
start=$(date +%s)
output=$(nix build ".#nixosTestConfigurations.${host}" --no-link -L 2>&1)
exit_code=$?
end=$(date +%s)
elapsed=$((end - start))

if [ $exit_code -eq 0 ]; then
echo "=== PASS: ${host} (${elapsed}s) ==="
RESULTS+=("PASS:${host}:${elapsed}s")
else
echo "=== FAIL: ${host} (exit code ${exit_code}, ${elapsed}s) ==="
RESULTS+=("FAIL:${host}:exit=${exit_code}:${elapsed}s")
fi
# Print last 20 lines of output for context
echo "--- Last 20 lines of ${host} output ---"
echo "${output}" | tail -n 20
echo "---"
done

echo ""
echo "============================================"
echo "=== SUMMARY ==="
echo "============================================"
for r in "${RESULTS[@]}"; do
echo "$r"
done
1 change: 1 addition & 0 deletions docs/src/SUMMARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
- [Adding an External Package](development/adding_an_external_package.md)
- [Declarative Gnome Dconf](development/declarative_gnome_dconf.md)
- [Using a Package/Module from a Fork](development/using_a_nix_package_or_nixos_module_from_a_separate_fork_of_nixpkgs.md)
- [VM Integration Tests](development/vm_integration_tests.md)

# Modules

Expand Down
Loading
Loading