🐛 Fix flaky hub validation of synthesized discovery rulesets; run unit tests in CI#98
Conversation
The tackle-hub target synthesizes discovery-rules/technology-usage rulesets from the hub's discovery-task tags, which have no kantra equivalent and are never compared for the hub target. Whether those rulesets survive output filtering depends on whether the discovery tasks finish before koncur snapshots application tags, so tests whose kantra-generated expected output lacks a technology-usage ruleset (e.g. nerd-dinner) fail or pass depending on task timing. Skip the unexpected-ruleset error for tackle-hub when the ruleset is one of the synthesized names and contains nothing but tags. Rulesets carrying real content (violations/insights/errors) still fail, and the missing-ruleset direction is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
No CI job ran go test, so the unit tests in pkg/config and pkg/targets have been uncompilable since the CustomRule refactor (konveyor#44) without anyone noticing. - Update rule-handling tests for the CustomRule model, and rewrite the prepareRules/prepareRulesForHub tests to exercise the real methods (including the hub's one-git-repo and no-mixed-rules error paths) instead of simulating removed logic - Stub the hub login probe in TestNewTackleHubTarget with httptest instead of hitting localhost:8080 (3 minutes of retries) - Add a unit-tests job (go vet + go test) to the PR CI workflow Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
📝 WalkthroughWalkthroughTest suites across pkg/config, pkg/targets, and pkg/validator are refactored to use structured config.CustomRule/GitRepoRule types instead of string-based analysis.Rules. validator.go adds logic to skip synthesized tag-only rulesets for tackle-hub targets. A new CI job runs go vet and go test. ChangesCustomRule refactor, hub validation, and CI unit tests
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yaml:
- Around line 10-25: The unit-tests workflow job is using overly broad default
permissions and is persisting the checkout token in git config. Update the
unit-tests job in the ci workflow to add a permissions block limited to
contents: read, and configure the actions/checkout step to disable credential
persistence with persist-credentials set to false. Use the unit-tests job and
the checkout step as the points to locate the change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bbd4a684-3879-4e18-942e-8cd67bb80f9e
📒 Files selected for processing (6)
.github/workflows/ci.yamlpkg/config/git_url_test.gopkg/targets/kantra_test.gopkg/targets/tackle_hub_test.gopkg/validator/validator.gopkg/validator/validator_test.go
Scope the job token to contents: read and skip credential persistence on checkout; the job only runs go vet and go test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
|
PR cherry-picked to branch release-0.10. Backport PR: #102 |
…t tests in CI (#98) (#102) * 🐛 Ignore hub-synthesized tag-only discovery rulesets in validation The tackle-hub target synthesizes discovery-rules/technology-usage rulesets from the hub's discovery-task tags, which have no kantra equivalent and are never compared for the hub target. Whether those rulesets survive output filtering depends on whether the discovery tasks finish before koncur snapshots application tags, so tests whose kantra-generated expected output lacks a technology-usage ruleset (e.g. nerd-dinner) fail or pass depending on task timing. Skip the unexpected-ruleset error for tackle-hub when the ruleset is one of the synthesized names and contains nothing but tags. Rulesets carrying real content (violations/insights/errors) still fail, and the missing-ruleset direction is unchanged. * 🌱 Fix stale unit tests and run them in CI No CI job ran go test, so the unit tests in pkg/config and pkg/targets have been uncompilable since the CustomRule refactor (#44) without anyone noticing. - Update rule-handling tests for the CustomRule model, and rewrite the prepareRules/prepareRulesForHub tests to exercise the real methods (including the hub's one-git-repo and no-mixed-rules error paths) instead of simulating removed logic - Stub the hub login probe in TestNewTackleHubTarget with httptest instead of hitting localhost:8080 (3 minutes of retries) - Add a unit-tests job (go vet + go test) to the PR CI workflow * 🌱 Restrict unit-tests CI job permissions Scope the job token to contents: read and skip credential persistence on checkout; the job only runs go vet and go test. --------- Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us> Signed-off-by: Cherry Picker <noreply@github.com> Co-authored-by: Fabian von Feilitzsch <fabian@fabianism.us> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Summary
Fixes the race reported by @mguetta1 where hub-target tests fail with
Unexpected ruleset found: technology-usagedepending on discovery-task timing (nerd-dinner, locally flaky while nightly stays green).Root cause: the tackle-hub target synthesizes
discovery-rules/technology-usagerulesets from the hub's discovery-task tags (no kantra equivalent, and tag contents are never compared for the hub target). koncur only waits on the analysis task, and empty rulesets are filtered before validation — so if tech-discovery finishes before the tag snapshot the synthesized ruleset survives with tags and trips the shared unexpected-ruleset check; if not, it's filtered and the test passes. Only C# is affected because the kantra-generated expected output for nerd-dinner has notechnology-usageruleset (all Java expected outputs do).Fix: the unexpected-ruleset check now skips
discovery-rules/technology-usagefor tackle-hub when the ruleset contains nothing but tags. Rulesets with real content (violations/insights/errors) still fail, and the missing-ruleset direction is unchanged. Historical hub outputs show insight reporting for tagging rules has varied across hub versions, so tolerating the fabricated tag-only ruleset is the only behavior correct under all observed variants (as opposed to dropping the tag collection, which would break Java tests on hubs that don't report tagging insights).Also: unit tests now run in CI
While verifying, we found no CI job runs
go test— the unit tests inpkg/configandpkg/targetshave been uncompilable since theCustomRulerefactor (#44, ~4 months) without anyone noticing.CustomRulemodel; rewroteprepareRules/prepareRulesForHubtests to exercise the real methods (including the hub's one-git-repo and no-mixed-rules error paths) instead of simulating removed logicTestNewTackleHubTargetnow stubs the login probe withhttptestinstead of retrying againstlocalhost:8080for 3 minutesunit-testsjob (go vet ./...+go test ./...) in the PR CI workflowTest plan
go vet ./...cleango test ./...green (pkg/targets down from 180s of network retries to <1s)TestValidateFiles_HubSynthesizedDiscoveryRulesetscovers the exact failure shape: tag-onlytechnology-usageabsent from expected passes for tackle-hub, still fails for kantra, and a synthesized-name ruleset with insights still fails🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Tests