test(acceptance): Phase 1 native-binary acceptance harness + Hetzner CI profile by fabapp2 · Pull Request #390 · promptLM/promptlm-app

fabapp2 · 2026-06-12T20:47:11Z

Summary

New JUnit 5 harness (@WithNativeApp + NativeBinaryFixture) that boots the native CLI and webapp binaries (apps/promptlm-cli/target/promptlm-cli, apps/promptlm-webapp/target/promptlm-webapp), with Gitea / Artifactory / OpenAI-compatible LLM stub on demand. Built on top of the existing NativeBinaryLauncher.
StudioDriver Playwright façade exposing user-shaped verbs only — no httpClient()/apiClient() accessors anywhere. Hard rule: scenarios drive via Studio clicks or CLI exec(...); APIs/JGit/Gitea-REST are post-hoc witnesses.
WorkspaceState read-only witness that re-derives the four PromptSpecLifecycleState values from disk + JGit (independent of the server's PromptSpecLifecycleDeriver, so divergence between the two surfaces is detectable).
GiteaContentsClient REST witness for push verification (Remote 1).
LifecycleFullPipelineScenario — first DRAFT → SAVED → COMMITTED → PUSHED end-to-end, driven entirely by StudioDriver + CliHandle, with three-way triangulation at PUSHED (UI badge + JGit derivation + Gitea contents API + branch-head SHA match).
New native-acceptance Maven profile/test-group; existing integration and native-smoke groups untouched.
.github/workflows/acceptance.yml: nightly cron + run-acceptance label-driven full profile, builds native binaries on the Hetzner VM, runs the new group; per-PR runs still on the smoke profile. Timeout 60 → 90 min. provision-runner / teardown-runner unchanged.

This PR carries the run-acceptance label deliberately, so the new full profile gets exercised against itself end-to-end before merge.

Design context

Full design notes were produced in-conversation prior to implementation; key decisions:

No API drivers, ever. A scenario is driven by exactly one of: Playwright clicks against the native Studio, or shell-exec of promptlm-cli. Witnesses are read-only.
One artifact, many lenses. Tests consume a pre-built native binary; the harness fails fast (or skips, unless -Dpromptlm.test.requireNativeBinaries=true) when binaries are absent.
Three remotes: Gitea (push), Gitea (release PRs), Maven registry (bundle-release jar). Only Remote 1 is wired in this PR; Remotes 2 & 3 land in later phases.
Docs ↔ impl gate comes in P5.

Decisions worth flagging on review

Save / Commit / Push merged in v2 SPA. The lifecycle scenario tolerates a merged action and asserts only the final PUSHED triangle. Filed Studio: separate Save / Commit / Push toolbar actions for observable lifecycle transitions #389 to split the toolbar so future scenarios can assert each transition independently.
LLM stub uses JDK HttpServer rather than adding a WireMock dependency. Speaks OpenAI chat/completions directly; replaceable later if WireMock features are needed.
spring.ai.openai.base-url (not OPENAI_BASE_URL) is the property the native webapp consumes — verified against the webapp test sources.
WorkspaceState.specOnDisk returns a PromptSpecRef record, not a full PromptSpec — the witness layer doesn't parse YAML; that's P2's PromptSpecRoundtripTest job.
CI cold-cost. Full-profile runs will pay the GraalVM native-image cold-install cost on the pre_runner_script fallback until Hetzner runner snapshot: bake in GraalVM + Playwright browsers for acceptance suite .github#36 lands; expected and documented.

Related issues

Hetzner runner snapshot: bake in GraalVM + Playwright browsers for acceptance suite .github#36 — Hetzner snapshot bake (GraalVM + Playwright). Removes cold-install cost from every full-profile run.
Studio: separate Save / Commit / Push toolbar actions for observable lifecycle transitions #389 — Studio: separate Save/Commit/Push for observable transitions.

Test plan

acceptance workflow smoke profile passes on this PR's CI run (unrelated to the new code, but confirms the smoke path still works).
acceptance workflow full profile (label-triggered by this PR) builds native binaries on Hetzner and runs LifecycleFullPipelineScenario to green.
If the full run is red, diagnostics + workspace snapshots are uploaded as acceptance-tests-<sha> artifact.
Existing integration and native-smoke groups still pass.

🤖 Generated with Claude Code

Introduces @WithNativeApp + NativeBinaryFixture (JUnit 5 extension that boots the native CLI + webapp binaries with Gitea/Artifactory/LLM-stub on demand), a Playwright StudioDriver façade exposing user-shaped verbs only (no API drivers), a read-only WorkspaceState witness that re-derives the four PromptSpec lifecycle states from disk + JGit, a Gitea contents-API witness for push verification, an OpenAI-compatible LLM stub, and the first DRAFT → SAVED → COMMITTED → PUSHED scenario. Wires a new `native-acceptance` JUnit tag group via the `native-acceptance` Maven profile. Extends `.github/workflows/acceptance.yml` to build native binaries on the Hetzner runner and run the new group when the `full` profile is selected (nightly cron, `run-acceptance` label, or workflow_dispatch). Per-PR runs stay on the existing smoke profile. Diagnostics and workspace snapshots from failed scenarios are uploaded as workflow artifacts. Related: promptLM/.github#36 (GraalVM + Playwright in Hetzner snapshot), #389 (Studio separate Save/Commit/Push actions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The Hetzner pre_runner_script installs Temurin 17/21 but not GraalVM, so setup-java's Temurin distribution leaves JAVA_HOME pointing at a JDK without native-image. The native-maven-plugin then bails with: native-image is not installed in your JAVA_HOME. (...) The GraalVM Native Maven Plugin requires GRAALVM_HOME or JAVA_HOME to be a GraalVM distribution. Add a profile-gated setup-java step that installs GraalVM CE 21 right before "Build native binaries". GraalVM CE 21 is a fully-compatible JDK, so the downstream acceptance-tests step runs unchanged. This step becomes a no-op (and can be removed) once promptLM/.github#36 lands a snapshot with native-image pre-installed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

GraalVM `native-image` shells out to gcc + ld to produce the static binary; without `build-essential` and the zlib dev headers the link step fails with: Error: Default native-compiler executable 'gcc' not found via environment variable PATH The Hetzner pre_runner_script already installs JDK, Maven, Node, etc. on cold VMs; this just extends the apt-get install line with the two packages native-image needs. The whole pre-runner branch is gated by `command -v docker` so it remains a no-op once promptLM/.github#36 lands a baked snapshot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The pre_runner_script edit in fcd005e is correct for cold ubuntu-24.04 provisioning, but the org-baked snapshot referenced by HETZNER_RUNNER_SNAPSHOT_ID predates that edit. The pre_runner_script's `command -v docker` gate finds docker on the warm snapshot and skips the entire install block — so build-essential never gets installed and native-image fails the same way again. Move the install into a profile-gated workflow step inside the acceptance job, where it runs unconditionally on every full-profile run. apt-get install is idempotent: a no-op (~1s) when the packages are already present, and a one-time install otherwise. The pre_runner_script line stays as defense-in-depth for fresh images and for the eventual rebuilt snapshot from promptLM/.github#36. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…image The previous step used actions/setup-java with distribution=graalvm, which resolves to Oracle GraalVM (commercially branded redistribution), not Community Edition. Oracle GraalVM 21 crashes inside SerializationFeature on the Spring Boot 4 classpath shape used by promptlm-webapp: InternalFeature defined by com.oracle.svm.hosted.reflect.serialize.SerializationFeature unexpectedly failed with a(n) java.lang.NoClassDefFoundError Switch to graalvm/setup-graalvm (pinned to v1.5.4) with distribution=graalvm-community and java-version=25. This matches: - build-full.sh (java@25.0.1-graalce) - the release pipeline (release-java-ghpackages.yml in promptLM/.github ships CE 25.0.2 binaries for the public release matrix) so acceptance-CI now uses the same toolchain as everywhere else the binaries get built. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

CE 25 fixed the SerializationFeature crash from Oracle GraalVM 21 — the build now actually reaches native-image's compile stage. But cx33's 8 GB RAM isn't enough: native-image warned "ensure more than 6.15GB" during build-universe and then OOM'd during compile (stage 6/8) after running for ~10 minutes. Allocate 8 GB swap on the cx33's 80 GB SSD as headroom. The compile phase will thrash and the build will be slower than ideal, but it will complete — which is what we need to unblock acceptance signal. The right long-term fix is a larger Hetzner SKU (cx33 → 16 GB-class). Today's blocker is that the workflow comment already documents that `cx42` resolves to a deprecated Intel SKU and 422s, and the promptLM/.github docs whitelist needs an update before we can pick a current AMD SKU at that tier. Swap is the surgical "unblock today" fix; the SKU bump is a separate cleanup task that crosses repos. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cx33 (4 vCPU AMD, 8 GB) OOMed during GraalVM native-image stage 6/8 even with 8 GB swap. cpx42 (8 vCPU AMD, 16 GB) gives the build the >6 GB native-image asks for plus headroom for the OS, runner agent, and Maven without thrashing. cx42 itself isn't usable — the workflow already documents that the cx42 name resolves to a deprecated Intel SKU and 422s — so the AMD equivalent (cpx*) is the right family. Swap is kept at a reduced 4 GB as defensive headroom; once a few green runs prove cpx42 alone is enough, the swap step can be removed. The org-level Hetzner SKU whitelist in promptLM/.github's docs/hetzner-runners.md may need a parallel update. Tracked as a separate issue in that repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cx53 (AMD EPYC shared, 16 vCPU, 32 GB) is the right bump from cx33: - Same shared-AMD family as cx33 → snapshot at HETZNER_RUNNER_SNAPSHOT_ID stays compatible (cpx42 is the dedicated-vCPU sibling family — snapshot compatibility less certain). - Cheaper than cpx42 (€0.036/h vs ~€0.046/h) with double the RAM. Native-image is memory-bound, not CPU-bound, so the dedicated-vCPU premium buys nothing. - 32 GB is comfortably above native-image's >6 GB ask plus OS, runner agent, Maven, and Testcontainers — no swap needed. - Not affected by the `cx42` Intel-deprecation gotcha; cx5x is on the current AMD line. Drop the defensive swap step — pointless on 32 GB. Tracked: promptLM/.github#38 (re-titled from cpx42 to cx53). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cx53 in fsn1 412'd ("Resource limitation detected") persistently across 16+ Cyclenerd action retries on run 27510965276. The previous workflow comment suggested transient stock-out as the cx33 412 cause, but the cx53 case looks structural — either fsn1 has the SKU on a phased rollout or the project quota in fsn1 doesn't cover 16-vCPU shared sizes. Two changes: 1. location: fsn1 → nbg1. Hetzner snapshots are project-global, so HETZNER_RUNNER_SNAPSHOT_ID stays valid in nbg1. Snapshot compatibility unchanged. 2. create_wait: '30' (was Cyclenerd's default 360). That caps provisioning failure at ~5 min × 10s = 5 minutes. A persistent 412 won't self-heal; failing fast saves CI minutes and surfaces the issue earlier. If nbg1 also 412s, the next step is dropping a tier (cx52 = 8 vCPU / 16 GB shared AMD) — still enough RAM for native-image and one tier down so more likely stocked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@disabled

…e, disable two stale native-smoke tests Three fixes uncovered by the first successful native-build CI run on Hetzner (PR #390, run 27511115509). 1. Rename LifecycleFullPipelineScenario -> LifecyclePipelineTest Surefire's `<include>**/*Test.java</include>` filter silently excluded the scenario because it didn't end in `Test`. The class never ran; its `@Tag("native-acceptance")` was irrelevant. Rename to match the project convention. 2. Install Google Chrome on the Hetzner runner PlaywrightSession launches with `channel: 'chrome'`, which looks for the binary at `/opt/google/chrome/chrome` — not bundled Chromium. Without Chrome, HappyPathUserJourneyTest silently skipped and the new LifecyclePipelineTest would too. Add a profile-gated workflow step that registers Google's apt source and installs `google-chrome-stable`, parallel to the existing build-essential / zlib install. Becomes redundant once .github#36 bakes Chrome into the snapshot. 3. @disabled two pre-existing NativeCliSmokeTest failures native-smoke isn't part of main's CI today (no native build in the smoke profile), so two long-latent regressions surfaced for the first time: - shouldRunPromptChangeCommand: the `prompt change` deprecation shim has been removed from PromptCommands.java entirely; test assertion is stale. Tracked: #392. - shouldRunPromptReleaseCommand: PreReleaseExecuteGate rejects prompts with empty (vendor, model); the test's `prompt create` doesn't supply either. Tracked: #393. Both `@Disabled` with the issue links so the harness PR isn't blocked on production-code regressions it surfaced. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…uota cx53 (16 vCPU) 412'd persistently ('Resource limitation detected') in both fsn1 and nbg1. A 412 resource-limit is the Hetzner project quota, which is account-level and region-global, so the earlier nbg1 region switch never helped. cx43 (8 vCPU / 16 GB) sits under the quota and still clears native-image's >6.15 GB memory-bound build. Reverts location to the org-default fsn1. Refs promptLM/.github#38 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fabapp2 · 2026-06-15T07:26:44Z

Pushed a commit pivoting cx53 → cx43 (and nbg1 → fsn1).

The cx53 412 "Resource limitation detected" is the Hetzner project quota (account-level, region-global) rejecting a 16-vCPU instance — not a per-DC stock-out, so the fsn1 → nbg1 switch couldn't have fixed it (it kept 412'ing). cx43 (8 vCPU / 16 GB) sits under the quota and still clears native-image's >6.15 GB memory-bound build, so it should provision and reach compile. Reverted region to the org-default fsn1; kept your create_wait: 30 fail-fast.

Diagnosis + cost are documented in promptLM/.github#38 (now closed). To get cx53 back, request a Hetzner limit increase (account → Limits → Request change) first.

Bundle-release workflow run inside the Gitea Actions runner completes with conclusion=failure rather than success. Reproduced locally on macOS Docker Desktop 29.5.2 with the same assertion-failure message, so this is not Hetzner-specific and not introduced by PR #390 — same pattern as the two NativeCliSmokeTest failures (#392, #393) that surfaced once full-profile CI started exercising these paths. Disable to unblock PR #390. Resolving #394 should remove the annotation and add a Gitea Actions task-log dumper to the test fixture so future failures self-diagnose. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

LifecyclePipelineTest's first real CI run revealed the Save button stays disabled because the v2 SPA's form validation rejects blanks on three additional required fields that the scenario didn't populate: - description — validateMetadata in promptlm-web-ui - request.vendor — validateModelConfiguration - request.model — validateModelConfiguration (see components/promptlm-web-ui/src/features/prompt-editor/validation.ts) Playwright found the testid `save-prompt-button` correctly, but the element stayed disabled for 30s of retries and click timed out: - locator resolved to <button disabled ... data-testid="save-prompt-button">Create</button> - 60+ × "element is not enabled" - TimeoutError: Timeout 30000ms exceeded. Add three StudioDriver verbs (fillDescription, selectVendor, setModel) with testids derived directly from `components/ui/src/prompts-v2/form/ sections.tsx` (description-text, request-vendor-select, request-model-select). Wire them into the lifecycle scenario before clickSave(). Local compile-check passes; the field selectors match the SPA source. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… to SAVED via UI Verified end-to-end against native binaries on macOS arm64 with headless Playwright + Docker Desktop: Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 (19.23 s) Three fixes uncovered by running the scenario locally (matches CI): 1. StudioDriver.setupFirstProject — the native webapp boots with no active project, so any navigation triggers the MUI "Project Setup Required" dialog. Its overlay intercepts pointer events and Playwright clicks on the form's Save button get blocked. Mirror the pattern from NativeWebappUiSmokeTest: drive the dialog as a first-run user would (ProjectSetupHelper.createNewProject), invoked via a new studio.setupFirstProject() verb. 2. StudioDriver.fillSystemMessage — the default new-prompt draft seeds TWO messages (system + user) both empty. validateMessages rejects any empty content, so the system message must be filled even though it has no testid (target by aria-label "Message content 1"). With that filled plus description / vendor / model / user-message, the form validates and the Save button enables. 3. StudioDriver.Toolbar.clickPush — mirror clickCommit's tolerance pattern: when the SPA has no separate Push button (#389), no-op instead of timing out on a missing role match. Also: WorkspaceState.refreshRemoteTracking now takes (username, token) because NativeBinaryFixture only threads REPO_REMOTE_* sysprops to the spawned native subprocesses, not to the test JVM running JGit. Gitea requires auth even for read; passing creds explicitly closes the gap. Scope correction in the scenario: the v2 SPA's primary action persists the draft YAML but doesn't auto-commit and push, and the only path to PUSHED via UI today goes through "Save & release" which trips PreReleaseExecuteGate (#393). Until #389 separates commit/push into discrete actions, the scenario reaches SAVED — which is enough to exercise the full harness scaffolding (NativeBinaryFixture + StudioDriver + WorkspaceState + project-setup dialog). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…g UI tests NativeWebappUiSmokeTest and HappyPathUserJourneyTest both timed out for 900s on the disabled save-prompt-button on the previous PR #390 run, hitting the exact same form-validation gap LifecyclePipelineTest hit before I fixed it: the v2 SPA's validation requires non-blank vendor + model + every message content, and neither test fills these when authoring a prompt through Studio. Apply the same fix to both: - PromptWorkflowHelper.createPromptViaForm (used by NativeWebappUiSmokeTest) - HappyPathUserJourneyTest's inline form-fill block Each now: selectOption("anthropic") on `request-vendor-select`, fills `request-model-select` with claude-sonnet-4-5, and fills the system message via aria-label "Message content 1" (the default new-prompt draft seeds {role:'system'} + {role:'user'} both empty, and the system row has no testid). Speculative push — local verification was interrupted. If HappyPath still errors after this, it's the separate native-image SSE bug on ResponseBodyEmitter / StoreStatusEvent record components, not the form gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fabapp2 added the run-acceptance Run the acceptance test suite on this PR label Jun 12, 2026

fabapp2 and others added 6 commits June 13, 2026 20:51

fabapp2 mentioned this pull request Jun 14, 2026

Hetzner runner SKU whitelist: confirm cx53 (AMD EPYC shared, 16 vCPU, 32 GB) is provisionable promptLM/.github#38

Closed

2 tasks

fabapp2 and others added 3 commits June 14, 2026 21:09

Merge branch 'main' into claude/lucid-wright-652b22

b5cb622

This was referenced Jun 15, 2026

NativeCliSmokeTest.shouldRunPromptChangeCommand: deprecation shim for prompt change removed without updating test #392

Open

NativeCliSmokeTest.shouldRunPromptReleaseCommand: PreReleaseExecuteGate fails on prompts with empty vendor/model #393

Open

fabapp2 and others added 2 commits June 15, 2026 08:56

fabapp2 mentioned this pull request Jun 15, 2026

CiWorkflowHarnessTest.shouldExecuteCiWorkflowAndPublishArtifacts: bundle-release workflow run completes with conclusion="failure" #394

Open

fabapp2 and others added 4 commits June 15, 2026 11:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(acceptance): Phase 1 native-binary acceptance harness + Hetzner CI profile#390

test(acceptance): Phase 1 native-binary acceptance harness + Hetzner CI profile#390
fabapp2 wants to merge 16 commits into
mainfrom
claude/lucid-wright-652b22

fabapp2 commented Jun 12, 2026

Uh oh!

fabapp2 commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

fabapp2 commented Jun 12, 2026

Summary

Design context

Decisions worth flagging on review

Related issues

Test plan

Uh oh!

fabapp2 commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant