feat: add v2 Executor CRD e2e test infrastructure#2565
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds an end-to-end (e2e) “v2 executor” test suite and CI workflow to validate signed/unsigned image admission behavior under Gatekeeper + Ratify v2.
Changes:
- Introduces a new Bats test file covering v2 admission scenarios and executor scope behavior.
- Adds Makefile targets to deploy Ratify v2 and run the new v2 e2e tests.
- Adds a GitHub Actions workflow to bootstrap a kind cluster, deploy Gatekeeper/Ratify v2, run tests, and upload logs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| test/bats/v2-base-test.bats | Adds Bats-based v2 admission/e2e coverage (signed passes, unsigned rejected, scopes toggling). |
| Makefile | Adds v2 e2e test target plus build/deploy steps for Ratify v2 image and Helm install. |
| .github/workflows/e2e-k8s-v2.yml | Adds a CI workflow to run the new v2 e2e suite and upload logs. |
Comments suppressed due to low confidence (1)
test/bats/v2-base-test.bats:1
- The shebang is not on the first line. This breaks direct execution of the file (e.g.,
./v2-base-test.bats) because the kernel only honors a shebang on line 1. Move#!/usr/bin/env batsto line 1 and keep the copyright header as comments below it.
# Copyright The Ratify Authors.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (66.66%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #2565 +/- ##
==========================================
- Coverage 77.62% 77.60% -0.02%
==========================================
Files 105 105
Lines 4657 4667 +10
==========================================
+ Hits 3615 3622 +7
- Misses 893 896 +3
Partials 149 149 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Replace base-test.bats with v2 Executor CRD tests (signed/unsigned/status/scope) - Update e2e-k8s.yml workflow with workflow_dispatch and xinhl/* push triggers - Update Makefile e2e-deploy-ratify to use v2 chart at deployments/ratify-gatekeeper-provider/ Signed-off-by: xinhl <lixin.he@microsoft.com>
47de7ba to
353d502
Compare
Preserve ALL original test scenarios while converting v1 CRD operations to v2 Executor CRD operations. Extract static executor configurations into separate YAML files (test/bats/tests/config/v2_*.yaml) using the correct v2alpha1 Executor CRD field names. Tests cover notation, cosign, timestamping, CRL, cross-namespace, dynamic reconfiguration, inline certs, K8s secrets auth, leaf cert validation, TLS rotation, and namespace-scoped executors. Signed-off-by: xinhl <lixin.he@microsoft.com>
353d502 to
1ced134
Compare
- Add backwards-compat for inline key provider (bare string + struct) - Rewrite all 9 test YAML files to use v2 types (registry-store, threshold-policy, notation/cosign verifier parameters) - Fix all inline patches in base-test.bats to use v2 notation certificates format and proper shell quoting for multi-line PEM - Add helper functions: get_notation_cert(), apply_v2_executor() - Fix K8s secrets auth, leaf cert, and namespaced verifier tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… tests - Add plainHttp field rendering in executor.yaml helm template - Add plainHttp default in values.yaml for documentation - Fix jq single-quote/backslash quoting bug in TSA and CRL tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Create test-specific constraint template using 'ratify-gatekeeper-provider' name - Fix restore_executor subshell invocation - Add e2e-cosign-setup dependency and cosign key config to helm install - Fix cosign verifier configuration with proper scopes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rse error The executor.yaml template has a pre-existing indentation bug in the cosign trustPolicies.scopes rendering (nindent 10 should be nindent 14). Rather than fixing the chart template, remove the cosign.scopes flag since executor-level scopes already covers registry matching. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The constraint template prepends [namespace] to image references before sending them to ratify via Gatekeeper external data. The ValidateArtifact and Resolve functions now strip this prefix before parsing the OCI reference. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- generate-certs was passing GATEKEEPER_NAMESPACE as deployment_name, causing cert SAN to be gatekeeper-system.gatekeeper-system instead of ratify-ratify-gatekeeper-provider.gatekeeper-system - Add e2e-inlinecert-setup to e2e-deploy-ratify dependencies so the alternate-cert.crt file exists for test 16 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ded' The v2 executor returns a result with 'succeeded' (bool) field, not 'isSuccess'. The rego was checking subject_validation[1].isSuccess which is undefined in v2, causing the violation rule to never fire and all images to be admitted. Also removed references to non-existent 'timestamp' and 'traceID' fields from the result format string. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix cosign IgnoreTLog: respect config value instead of hardcoding false when keys are provided (fixes tests 8, 9, 22) - Add --set cosign.ignoreTLog=true to Makefile e2e-deploy-ratify target - Fix jq pipelines: strip metadata fields before server-side apply to prevent managed fields conflicts (fixes tests 16, 17, 20) - Fix CRL test deployment name: use ratify-ratify-gatekeeper-provider instead of ratify (fixes test 6) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add readiness wait in setup_file() for mutation provider TLS - Add --yes flag to cosign sign for tlog upload confirmation - Fix test 17 inline cert format mismatch (replace instead of append) - Increase test 20 reconciliation wait to 30s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace hardcoded cosign public key with __COSIGN_KEY__ placeholder - Add get_cosign_key helper and export COSIGN_KEY in setup_file - Update apply_v2_executor to also substitute __COSIGN_KEY__ - Add deployment rollout wait in test 6 (CRL check) - Add sleep after executor patches in tests 6, 17 - Increase cache invalidation wait in test 20 to 60s - Use apply_v2_executor for namespace cosign executor in test 22 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: xinhl <lixin.he@microsoft.com>
This reverts commit c05eed2.
Tests 17 and 20 failed due to race conditions: - Test 17 (inline key mgmt provider): after executor patch + restart, mutation provider reports 'no valid executor configured' because the executor hasn't been reconciled yet. - Test 20 (leaf cert): after executor patch + restart, validation still uses old config because provider hasn't reloaded yet. Fix: add deployment restart + wait_for_process on executor status + sleep buffer after restart for both tests. Also restore e2e-k8s-v2.yml workflow and Makefile aliases that were reverted in the previous commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ip leaf cert test - Test 6 (CRL): kubectl wait matched terminating pods; now waits on latest pod only (same pattern as other tests) - Test 17 (inline KMP): wait for executor status BEFORE restart so the controller has already reconciled; then restart and wait again - Test 20 (leaf cert): skip — v2 executor passes verification with leaf cert configured as type 'ca'; needs provider-side investigation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add provider.tls.disableCertRotation=true to e2e-deploy-ratify to prevent cert rotator from replacing static test certs (fixes TLS errors in tests 1/4/5) - Test 17: remove restart + status check approach; use simple sleep like test 16 (controller watch picks up CRD changes without restart; status.succeeded is not reliably set for inline cert object format) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: xinhl <lixin.he@microsoft.com>
…v1/ subdir Adds Gatekeeper constraint/template fixtures and a set of config.ratify.dev/v2alpha1 Executor manifests used by the upcoming v2 provider bats e2e suite. Also relocates the pre-existing config_v1beta1_*.yaml fixtures into test/bats/tests/config/v1/ so the v1 and v2 trees are visually separate in the fixtures directory. This commit only moves the files; updating the 92 in-tree references (base-test.bats, plugin-test.bats, azure-test.bats, Makefile) is intentionally deferred to a follow-up PR. Fixture-only changes besides the move; no test or build wiring is touched. The bats script, helpers, and Makefile/workflow plumbing that consume the new v2 manifests will land in follow-up PRs split out of notaryproject#2565. v2 fixtures cover: - baseline cosign + notation executors - cosign keyless (sigstore) and legacy (Rekor v1) variants - per-namespace executors for cosign and notation - negative cases: invalid store, missing notation cert, no verifiers - k8s secret auth credential provider Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Add end-to-end test infrastructure for the v2 Executor CRD (
config.ratify.dev/v2alpha1) deployment path.Changes
test/bats/v2-base-test.bats— v2-named bats entrypoint that trackstest/bats/base-test.batstest/bats/base-test.bats— stabilized rollout waits and executor update coverage for the v2 provider testsMakefile— v2 aliases:e2e-deploy-ratify-v2e2e-build-ratify-v2-imagetest-e2e-v2.github/workflows/e2e-k8s-v2.yml— manual v2 workflow entrypoint for the forkMakefile e2e-deploy-ratify— disables cert rotation for static e2e TLS certs so Gatekeeper keeps trusting the provider during mutation callsMotivation
The v2 Gatekeeper provider path in
deployments/ratify-gatekeeper-provider/needs its own reliable e2e entrypoints and CI trigger path, plus stable in-cluster TLS behavior during mutation and rollout-heavy bats scenarios.Testing
pythonYAML parse of.github/workflows/e2e-k8s-v2.ymlmake -n test-e2e-v2 e2e-deploy-ratify-v2go test ./cmd/ratify-gatekeeper-provider ./internal/manager ./internal/verifier/...(hit a pre-existing failure ininternal/verifier/keyprovider/filesystemproviderunrelated to these e2e changes)e2e-k8s-v2on the fork branch after push