From 98f1de803f90a5e8a32af2addad1611076e4261b Mon Sep 17 00:00:00 2001 From: Guillaume Nodet Date: Fri, 10 Apr 2026 01:03:26 +0200 Subject: [PATCH 1/4] ci: use Scalpel skip-tests mode for single-invocation CI builds Replace the multi-step CI pipeline (regen.sh build + incremental-build.sh module detection + separate Maven test invocation) with a single Maven invocation using Scalpel 0.2.0's skip-tests mode. Scalpel now builds all modules but only runs tests on affected ones, eliminating ~500 lines of grep-based module detection and threshold logic. The incremental-build.sh script becomes a post-build comment generator that reads Scalpel's JSON report. Changes: - Update Scalpel extension3:0.1.0 to extension:0.2.0 - Add Scalpel skip-tests flags to regen.sh (CI-only, no-op on dev machines) - Add --skip-tests and --coverage arguments to regen.sh - Rewrite incremental-build.sh as post-build comment generator (~200 lines) - Simplify sonar-build.yml to use regen.sh --coverage - Move skip-tests label check from script to workflow step - Update CI-ARCHITECTURE.md for new architecture Co-Authored-By: Claude Opus 4.6 --- .github/CI-ARCHITECTURE.md | 104 ++- .github/actions/incremental-build/action.yaml | 35 +- .../incremental-build/incremental-build.sh | 724 +++--------------- .github/workflows/pr-build-main.yml | 49 +- .github/workflows/sonar-build.yml | 19 +- .mvn/extensions.xml | 4 +- etc/scripts/regen.sh | 37 +- 7 files changed, 248 insertions(+), 724 deletions(-) diff --git a/.github/CI-ARCHITECTURE.md b/.github/CI-ARCHITECTURE.md index 4672dcd5a9c3c..c6ade2526f912 100644 --- a/.github/CI-ARCHITECTURE.md +++ b/.github/CI-ARCHITECTURE.md @@ -11,11 +11,8 @@ PR opened/updated │ ├──► pr-build-main.yml (Build and test) │ │ - │ ├── regen.sh (full build, no tests) - │ ├── incremental-build (test affected modules) - │ │ ├── File-path analysis - │ │ ├── POM dependency analysis - │ │ └── Extra modules (/component-test) + │ ├── regen.sh (build + test via Scalpel skip-tests) + │ ├── generate test report (post-build comment) │ │ │ └──► pr-test-commenter.yml (post unified comment) │ @@ -36,10 +33,11 @@ PR comment: /component-test kafka http - **Trigger**: `pull_request` (main branch), `workflow_dispatch` - **Matrix**: JDK 17, 21, 25 (25 is experimental) - **Steps**: - 1. Full build via `regen.sh` (`mvn install -DskipTests -Pregen`) - 2. Check for uncommitted generated files - 3. Run incremental tests (only affected modules) - 4. Upload test comment as artifact + 1. Check `skip-tests` label + 2. Build and test via `regen.sh` (single Maven invocation with Scalpel) + 3. Check for uncommitted generated files + 4. Generate test report from Scalpel's JSON output + 5. Upload test comment as artifact - **Inputs** (workflow_dispatch): `pr_number`, `pr_ref`, `extra_modules`, `skip_full_build` ### `pr-test-commenter.yml` — Post CI test comment @@ -70,8 +68,8 @@ PR comment: /component-test kafka http - **Status**: Temporarily disabled (INFRA-27808 — SonarCloud quality gate adjustment pending) - **Trigger**: `pull_request` (main branch) → `workflow_run` on SonarBuild completion -- **Why two workflows**: `sonar-build.yml` runs in PR context (builds with JaCoCo coverage on core modules, uploads compiled classes artifact), `sonar-scan.yml` runs via `workflow_run` with secrets access to run the Sonar scanner and post results -- **Coverage scope**: Currently limited to core modules (`camel-api`, `camel-core`, etc.) and `coverage` aggregator. Component coverage planned for future integration with `incremental-build.sh` module detection +- **Why two workflows**: `sonar-build.yml` runs in PR context (builds with JaCoCo coverage via Scalpel, uploads compiled classes artifact), `sonar-scan.yml` runs via `workflow_run` with secrets access to run the Sonar scanner and post results +- **Coverage scope**: Scalpel dynamically detects affected modules and runs tests with JaCoCo coverage on them — no hardcoded module list ### Other workflows @@ -87,22 +85,19 @@ PR comment: /component-test kafka http ### `incremental-build` -The core test runner. Determines which modules to test using: +Post-build test report generator. Reads Scalpel's JSON report (`target/scalpel-report.json`) and generates a PR comment showing: -1. **File-path analysis**: Maps changed files to Maven modules -2. **POM dependency analysis** (dual detection): - - **Grep-based**: For `parent/pom.xml` changes, detects property changes and finds modules that explicitly reference the affected properties via `${property}` in their `pom.xml` files - - **Scalpel-based**: Uses [Maveniverse Scalpel](https://github.com/maveniverse/scalpel) (Maven extension) for effective POM model comparison — catches managed dependencies, plugin version changes, BOM imports, and transitive dependency impacts that the grep approach misses -3. **Extra modules**: Additional modules passed via `/component-test` - -Both detection methods run in parallel. Their results are merged (union), deduplicated, and tested. If Scalpel fails (build error, runtime error), the script falls back to grep-only with no regression. +1. **Directly affected modules** with detection reasons (`SOURCE_CHANGE`, `POM_CHANGE`, `TRANSITIVE_DEPENDENCY`, `MANAGED_PLUGIN`, `TEST_CHANGE`) +2. **Changed properties, managed dependencies, managed plugins** +3. **Downstream modules** tested as dependents (collapsible) +4. **Upstream modules** compiled but with tests skipped (collapsible) +5. **Extra modules** from `/component-test` The script also: - Detects tests disabled in CI (`@DisabledIfSystemProperty(named = "ci.env.name")`) -- Applies an exclusion list for generated/meta modules - Checks for excluded modules with associated integration tests (via `manual-it-mapping.txt`) and advises contributors to run them manually -- Generates a unified PR comment with all test information +- Parses test failures from surefire/failsafe reports ### `install-mvnd` @@ -112,57 +107,56 @@ Installs the Maven Daemon (mvnd) for faster builds. Installs system packages required for the build. -## PR Labels +## Scalpel: Automatic Test Selection -| Label | Effect | -| --- | --- | -| `skip-tests` | Skip all tests | -| `test-dependents` | Force testing dependent modules even if threshold exceeded | +[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) is a Maven core extension that detects which modules are affected by a PR's changes and controls test execution accordingly. -## CI Environment - -The CI sets `-Dci.env.name=github.com` via `MVND_OPTS` (in `install-mvnd`). Tests can use `@DisabledIfSystemProperty(named = "ci.env.name")` to skip flaky tests in CI. The test comment warns about these skipped tests. +### How it works -## POM Dependency Detection: Dual Approach +Scalpel is configured in `.mvn/extensions.xml` and runs automatically during Maven builds. In CI, it operates in **skip-tests** mode: all modules are compiled, but tests are only executed on affected modules. -### Grep-based detection (legacy) +Detection capabilities: +- **Source changes**: Files changed in a module's directory +- **POM changes**: Property, dependency, and plugin changes in any `pom.xml` +- **Transitive dependencies**: Changes that propagate through the dependency graph +- **Managed dependencies**: Version changes via `` (even without explicit `` in child modules) +- **Managed plugins**: Plugin version changes via `` +- **Property indirection**: `${property}` references resolved transitively -The grep approach searches for `${property-name}` references in module `pom.xml` files. It has known limitations: +### Configuration -1. **Managed dependencies without explicit ``** — Modules inheriting versions via `` without declaring `${property}` are missed. -2. **Maven plugin version changes** — Plugin version properties consumed in `parent/pom.xml` via `` are invisible to child modules. -3. **BOM imports** — Modules using artifacts from a BOM are not linked to the BOM version property. -4. **Transitive dependency changes** — Only direct property references are detected. -5. **Non-property version changes** — Structural `` edits without property substitution are not caught. +Scalpel flags are set in `etc/scripts/regen.sh` when running in CI (`GITHUB_ACTIONS=true`): -### Scalpel-based detection (new) +| Flag | Value | Purpose | +| --- | --- | --- | +| `scalpel.mode` | `skip-tests` | Build all, test only affected | +| `scalpel.skipTestsForUpstream` | `true` | Don't test upstream-only modules | +| `scalpel.fetchBaseBranch` | `true` | Auto-fetch base branch in shallow clones | +| `scalpel.fullBuildTriggers` | *(empty)* | Override `.mvn/**` default | +| `scalpel.reportFile` | `target/scalpel-report.json` | JSON report for PR comment | +| `scalpel.impactedLog` | `target/scalpel-impacted.txt` | Simple module path list | -[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) is a Maven core extension that compares effective POM models between the base branch and the PR. It resolves all 5 grep limitations by: +### Developer machines -- Reading old POM files from the merge-base commit (via JGit) -- Comparing properties, managed dependencies, and managed plugins between old and new POMs -- Resolving the full transitive dependency graph to find all affected modules -- Detecting plugin version changes via `project.getBuildPlugins()` comparison +On developer machines, Scalpel is a no-op: without `GITHUB_BASE_REF` (set by GitHub Actions for PRs), no base branch is detected and Scalpel returns immediately. Zero impact on local builds. -Scalpel runs in **report mode** (`-Dscalpel.mode=report`), writing a JSON report to `target/scalpel-report.json` without modifying the Maven reactor. The report includes affected modules with reasons (`SOURCE_CHANGE`, `POM_CHANGE`, `TRANSITIVE_DEPENDENCY`, `MANAGED_PLUGIN`). +### Fail-safe -### Dual-detection strategy +If Scalpel encounters an error, `failSafe=true` (the default) causes it to fall back to a full build — all tests run. This means Scalpel errors never cause false negatives (missed tests). -Both methods run in parallel. Results are merged (union) before testing. This lets us: +## PR Labels -1. **Validate Scalpel** — Compare what each method detects across many PRs -2. **No regression** — If Scalpel fails, grep results are still used -3. **Gradual migration** — Once Scalpel is validated, grep can be removed +| Label | Effect | +| --- | --- | +| `incremental-skip-tests` | Skip all tests (pass `--skip-tests` to regen.sh) | -Scalpel is configured permanently in `.mvn/extensions.xml` (version `0.1.0`). On developer machines it is a no-op — without CI environment variables (`GITHUB_BASE_REF`), no base branch is detected and Scalpel returns immediately. The `mvn validate` with report mode adds ~60-90 seconds in CI. +## CI Environment -Note: the script overrides `fullBuildTriggers` to empty (`-Dscalpel.fullBuildTriggers=`) because Scalpel's default (`.mvn/**`) would trigger a full build whenever `.mvn/extensions.xml` itself changes (e.g., Dependabot bumping Scalpel). +The CI sets `-Dci.env.name=github.com` via `MVND_OPTS` (in `install-mvnd`). Tests can use `@DisabledIfSystemProperty(named = "ci.env.name")` to skip flaky tests in CI. The test comment warns about these skipped tests. ## Manual Integration Test Advisories -Some modules are excluded from CI's `-amd` expansion (the `EXCLUSION_LIST`) because they are generated code, meta-modules, or expensive integration test suites. When a contributor changes one of these modules, CI cannot automatically test all downstream effects. - -The file `manual-it-mapping.txt` (co-located with the incremental build script) maps source modules to their associated integration test suites. When a changed module has a mapping entry, CI posts an advisory in the PR comment: +Some integration test suites are excluded from CI. When a contributor changes a module with a mapping entry in `manual-it-mapping.txt`, CI posts an advisory in the PR comment: > You modified `dsl/camel-jbang/camel-jbang-core`. The related integration tests in `dsl/camel-jbang/camel-jbang-it` are excluded from CI. Consider running them manually: > @@ -184,4 +178,4 @@ All non-experimental JDK matrix entries (17, 21) upload the CI comment artifact PR comments use HTML markers for upsert (create-or-update) behavior: -- `` — Unified test summary comment \ No newline at end of file +- `` — Unified test summary comment diff --git a/.github/actions/incremental-build/action.yaml b/.github/actions/incremental-build/action.yaml index 6ca313f40801e..76f23c069a192 100644 --- a/.github/actions/incremental-build/action.yaml +++ b/.github/actions/incremental-build/action.yaml @@ -15,40 +15,23 @@ # limitations under the License. # -name: "Incremental Test Runner" -description: "Test only affected projects, using file-path analysis and POM dependency detection" +name: "Test Report Generator" +description: "Generate PR comment from Scalpel's change detection report" inputs: - pr-id: - description: 'Id of the pull request (optional for push builds)' + extra-modules: + description: 'Additional modules tested (comma-separated paths, e.g. from /component-test)' required: false default: '' - github-token: - description: 'The github token to access to the API' - required: false - skip-mvnd-install: - description: 'Flag indicating whether the installation of mvnd should be skipped' + build-failed: + description: 'Whether the build step failed (true/false)' required: false default: 'false' - github-repo: - description: 'The GitHub repository name (example, apache/camel)' - required: false - default: ${{ github.repository }} - extra-modules: - description: 'Additional modules to test (comma-separated paths, e.g. from /component-test)' - required: false - default: '' runs: using: "composite" steps: - - id: install-mvnd - uses: apache/camel/.github/actions/install-mvnd@main - with: - dry-run: ${{ inputs.skip-mvnd-install }} - - name: maven test + - name: Generate test report env: - GITHUB_TOKEN: ${{ inputs.github-token }} - PR_ID: ${{ inputs.pr-id }} - GITHUB_REPO: ${{ inputs.github-repo }} EXTRA_MODULES: ${{ inputs.extra-modules }} + BUILD_FAILED: ${{ inputs.build-failed }} shell: bash - run: ${{ github.action_path }}/incremental-build.sh ${{ steps.install-mvnd.outputs.mvnd-dir }}/mvnd "$PR_ID" "$GITHUB_REPO" "$EXTRA_MODULES" + run: ${{ github.action_path }}/incremental-build.sh "$EXTRA_MODULES" diff --git a/.github/actions/incremental-build/incremental-build.sh b/.github/actions/incremental-build/incremental-build.sh index d2f01ae420a45..ba5ec423f3510 100755 --- a/.github/actions/incremental-build/incremental-build.sh +++ b/.github/actions/incremental-build/incremental-build.sh @@ -15,271 +15,25 @@ # limitations under the License. # -# Incremental test runner for Apache Camel PRs. +# Post-build comment generator for Apache Camel PRs. # -# Determines which modules to test by: -# 1. File-path analysis: maps changed files to their Maven modules -# 2. POM dependency analysis (dual detection): -# a. Grep-based: detects property changes in parent/pom.xml and finds -# modules that explicitly reference the affected properties -# b. Scalpel-based: uses Maveniverse Scalpel extension for effective POM -# model comparison — catches managed deps, plugin changes, BOM imports, -# and transitive dependency impacts that grep misses +# Reads Scalpel's JSON report (target/scalpel-report.json) to generate +# a PR comment showing which modules were tested and why. # -# All sets of affected modules are merged and deduplicated before testing. +# This script runs AFTER the build+test step, which uses Scalpel's +# skip-tests mode to build all modules and test only affected ones. +# See https://github.com/maveniverse/scalpel set -euo pipefail -echo "Using MVND_OPTS=$MVND_OPTS" - -maxNumberOfTestableProjects=50 - -# Modules excluded from targeted testing (generated code, meta-modules, etc.) -EXCLUSION_LIST="!:camel-allcomponents,!:dummy-component,!:camel-catalog,!:camel-catalog-console,!:camel-catalog-lucene,!:camel-catalog-maven,!:camel-catalog-suggest,!:camel-route-parser,!:camel-csimple-maven-plugin,!:camel-report-maven-plugin,!:camel-endpointdsl,!:camel-componentdsl,!:camel-endpointdsl-support,!:camel-yaml-dsl,!:camel-kamelet-main,!:camel-yaml-dsl-deserializers,!:camel-yaml-dsl-maven-plugin,!:camel-jbang-core,!:camel-jbang-main,!:camel-jbang-plugin-generate,!:camel-jbang-plugin-edit,!:camel-jbang-plugin-kubernetes,!:camel-jbang-plugin-test,!:camel-launcher,!:camel-jbang-it,!:camel-itest,!:docs,!:apache-camel,!:coverage" - # ── Utility functions ────────────────────────────────────────────────── -# Walk up from a file path to find the nearest directory containing a pom.xml -findProjectRoot() { - local path=${1} - while [[ "$path" != "." ]]; do - if [[ ! -e "$path/pom.xml" ]]; then - path=$(dirname "$path") - elif [[ $(dirname "$path") == */src/it ]]; then - path=$(dirname "$(dirname "$path")") - else - break - fi - done - echo "$path" -} - -# Check whether a PR label exists -hasLabel() { - local issueNumber=${1} - local label="incremental-${2}" - local repository=${3} - curl -s \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: 2022-11-28" \ - "https://api.github.com/repos/${repository}/issues/${issueNumber}/labels" | jq -r '.[].name' | { grep -c "$label" || true; } -} - -# Fetch the PR diff from the GitHub API. Returns the full unified diff. -fetchDiff() { - local prId="$1" - local repository="$2" - - local diff_output - diff_output=$(curl -s -w "\n%{http_code}" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "Accept: application/vnd.github.v3.diff" \ - "https://api.github.com/repos/${repository}/pulls/${prId}") - - local http_code - http_code=$(echo "$diff_output" | tail -n 1) - local diff_body - diff_body=$(echo "$diff_output" | sed '$d') - - if [[ "$http_code" -lt 200 || "$http_code" -ge 300 || -z "$diff_body" ]]; then - echo "WARNING: Failed to fetch PR diff (HTTP $http_code). Falling back to full build." >&2 - return - fi - echo "$diff_body" -} - -# ── POM dependency analysis (previously detect-dependencies) ─────────── -# -# Uses the pre-#22022 approach: grep for ${property-name} references in -# module pom.xml files. The Maveniverse Toolbox approach (introduced in -# #22022, reverted in #22279) is intentionally not used here. See -# CI-ARCHITECTURE.md for known limitations of the grep approach. - -# Extract the diff section for a specific pom.xml file from the full diff -extractPomDiff() { - local diff_body="$1" - local pom_path="$2" - - echo "$diff_body" | awk -v target="a/${pom_path}" ' - /^diff --git/ && found { exit } - /^diff --git/ && index($0, target) { found=1 } - found { print } - ' -} - -# Detect which properties changed in a pom.xml diff. -# Returns one property name per line. -# Filters out structural XML elements (groupId, artifactId, version, etc.) -# to only return actual property names (e.g. openai-java-version). -detectChangedProperties() { - local diff_content="$1" - - # Known structural POM elements that are NOT property names - local structural_elements="groupId|artifactId|version|scope|type|classifier|optional|systemPath|exclusions|exclusion|dependency|dependencies|dependencyManagement|parent|modules|module|packaging|name|description|url|relativePath" - - echo "$diff_content" | \ - grep -E '^[+-][[:space:]]*<[^>]+>[^<]*]+>' | \ - grep -vE '^\+\+\+|^---' | \ - sed -E 's/^[+-][[:space:]]*<([^>]+)>.*/\1/' | \ - grep -vE "^(${structural_elements})$" | \ - sort -u || true -} - -# Find modules that reference a property in their pom.xml. -# Searches pom.xml files under catalog/, components/, core/, dsl/ for -# ${property_name} references and extracts the module's artifactId. -# Adds discovered artifactIds to the dep_module_ids variable -# (which must be declared in the caller). -findAffectedModules() { - local property="$1" - - local matches - matches=$(grep -rl "\${${property}}" --include="pom.xml" . 2>/dev/null | \ - grep -v "^\./parent/pom.xml" | \ - grep -v "/target/" || true) - - if [ -z "$matches" ]; then - return - fi - - while read -r pom_file; do - [ -z "$pom_file" ] && continue - - # Only consider catalog, components, core, dsl paths (same as original detect-test.sh) - if [[ "$pom_file" == */catalog/* ]] || \ - [[ "$pom_file" == */components/* ]] || \ - [[ "$pom_file" == */core/* ]] || \ - ([[ "$pom_file" == */dsl/* ]] && [[ "$pom_file" != */dsl/camel-jbang* ]]); then - local mod_artifact - mod_artifact=$(sed -n '//,/<\/parent>/!{ s/.*\([^<]*\)<\/artifactId>.*/\1/p }' "$pom_file" | head -1) - if [ -n "$mod_artifact" ] && ! echo ",$dep_module_ids," | grep -q ",:${mod_artifact},"; then - echo " Property '\${${property}}' referenced by: $mod_artifact" - dep_module_ids="${dep_module_ids:+${dep_module_ids},}:${mod_artifact}" - fi - fi - done <<< "$matches" -} - -# Analyze pom.xml changes to find affected modules via property grep. -# Adds discovered module artifactIds to the dep_module_ids variable -# (which must be declared in the caller). -analyzePomDependencies() { - local diff_body="$1" - local pom_path="$2" # e.g. "parent/pom.xml" or "components/camel-foo/pom.xml" - - local pom_diff - pom_diff=$(extractPomDiff "$diff_body" "$pom_path") - if [ -z "$pom_diff" ]; then - return - fi - - local changed_props - changed_props=$(detectChangedProperties "$pom_diff") - if [ -z "$changed_props" ]; then - return - fi - - echo " Property changes detected in ${pom_path}:" - echo "$changed_props" | while read -r p; do echo " - $p"; done - - while read -r prop; do - [ -z "$prop" ] && continue - findAffectedModules "$prop" - done <<< "$changed_props" -} - -# ── POM dependency analysis via Scalpel (parallel) ───────────────────── -# -# Uses Maveniverse Scalpel (Maven extension) for effective POM model -# comparison. Detects changed properties, managed dependencies, managed -# plugins, and transitive dependency impacts that the grep approach misses. -# Runs alongside grep — results are merged (union) for testing. -# See https://github.com/maveniverse/scalpel - -# Run Scalpel in report mode to detect modules affected by POM changes. -# Sets caller-visible variables: scalpel_module_ids, scalpel_module_paths, -# scalpel_props, scalpel_managed_deps, scalpel_managed_plugins -runScalpelDetection() { - echo " Running Scalpel change detection..." - - # Ensure sufficient git history for JGit merge-base detection - # (CI uses shallow clones; Scalpel needs to find the merge base) - git fetch origin main:refs/remotes/origin/main --depth=200 2>/dev/null || true - git fetch --deepen=200 2>/dev/null || true - - # Scalpel is permanently configured in .mvn/extensions.xml. - # On developer machines it's a no-op (no GITHUB_BASE_REF → no base branch detected). - # Run Maven validate with Scalpel in report mode: - # - mode=report: write JSON report without trimming the reactor - # - fullBuildTriggers="": override .mvn/** default (Scalpel lives in .mvn/extensions.xml) - # - alsoMake/alsoMakeDependents=false: we only want directly affected modules - # (our script handles -amd expansion separately) - local scalpel_args="-Dscalpel.mode=report -Dscalpel.fullBuildTriggers= -Dscalpel.alsoMake=false -Dscalpel.alsoMakeDependents=false" - # For workflow_dispatch, GITHUB_BASE_REF may not be set - if [ -z "${GITHUB_BASE_REF:-}" ]; then - scalpel_args="$scalpel_args -Dscalpel.baseBranch=origin/main" - fi - - echo " Scalpel: running mvn validate (report mode)..." - ./mvnw -B -q validate $scalpel_args -l /tmp/scalpel-validate.log 2>/dev/null || { - echo " WARNING: Scalpel detection failed (exit $?), skipping" - grep -i "scalpel" /tmp/scalpel-validate.log 2>/dev/null | head -5 || true - return - } - - # Parse the Scalpel report - local report="target/scalpel-report.json" - if [ ! -f "$report" ]; then - echo " WARNING: Scalpel report not found at $report" - grep -i "scalpel" /tmp/scalpel-validate.log 2>/dev/null | head -5 || true - return - fi - - # Check if full build was triggered - local full_build - full_build=$(jq -r '.fullBuildTriggered' "$report") - if [ "$full_build" = "true" ]; then - local trigger_file - trigger_file=$(jq -r '.triggerFile // "unknown"' "$report") - echo " Scalpel: Full build triggered by change to $trigger_file" - return - fi - - # Extract affected module artifactIds (colon-prefixed for Maven -pl compatibility) - scalpel_module_ids=$(jq -r '.affectedModules[].artifactId' "$report" 2>/dev/null | sort -u | sed 's/^/:/' | tr '\n' ',' | sed 's/,$//' || true) - scalpel_module_paths=$(jq -r '.affectedModules[].path' "$report" 2>/dev/null | sort -u | tr '\n' ',' | sed 's/,$//' || true) - scalpel_props=$(jq -r '(.changedProperties // []) | if length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true) - scalpel_managed_deps=$(jq -r '(.changedManagedDependencies // []) | if length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true) - scalpel_managed_plugins=$(jq -r '(.changedManagedPlugins // []) | if length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true) - - local mod_count - mod_count=$(jq '.affectedModules | length' "$report" 2>/dev/null || echo "0") - echo " Scalpel detected $mod_count affected modules" - if [ -n "$scalpel_props" ]; then - echo " Changed properties: $scalpel_props" - fi - if [ -n "$scalpel_managed_deps" ]; then - echo " Changed managed deps: $scalpel_managed_deps" - fi - if [ -n "$scalpel_managed_plugins" ]; then - echo " Changed managed plugins: $scalpel_managed_plugins" - fi -} - -# ── Disabled-test detection ───────────────────────────────────────────── - # Scan tested modules for @DisabledIfSystemProperty(named = "ci.env.name") # and return a markdown warning listing affected files. detectDisabledTests() { - local final_pl="$1" local skipped="" - for mod_path in $(echo "$final_pl" | tr ',' '\n'); do - # Skip artifactId-style references (e.g. :camel-openai) — only scan paths - if [[ "$mod_path" == :* ]]; then - continue - fi + for mod_path in $@; do if [ -d "$mod_path" ]; then local matches matches=$(grep -rl 'DisabledIfSystemProperty' "$mod_path" --include="*.java" 2>/dev/null \ @@ -300,8 +54,9 @@ detectDisabledTests() { # Check if changed modules have associated integration tests excluded from CI. # Reads manual-it-mapping.txt and appends advisories to the PR comment. checkManualItTests() { - local final_pl="$1" - local comment_file="$2" + local comment_file="$1" + shift + local module_paths="$@" local script_dir script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" local mapping_file="${script_dir}/manual-it-mapping.txt" @@ -319,8 +74,8 @@ checkManualItTests() { it_module="${it_module// /}" command="${command#"${command%%[![:space:]]*}"}" - # Check if any module in final_pl matches this source_id - for module_path in $(echo "$final_pl" | tr ',' '\n'); do + # Check if any module path matches this source_id + for module_path in $module_paths; do if [[ "$(basename "$module_path")" == "$source_id" ]]; then it_commands["$it_module"]="$command" it_sources["$it_module"]="${it_sources[$it_module]:-}${it_sources[$it_module]:+, }\`${module_path}\`" @@ -346,63 +101,80 @@ checkManualItTests() { writeComment() { local comment_file="$1" - local pl="$2" - local dep_ids="$3" - local changed_props_summary="$4" - local testedDependents="$5" - local extra_modules="$6" - local managed_deps_summary="${7:-}" - local managed_plugins_summary="${8:-}" + local report="$2" + local extra_modules="$3" echo "" > "$comment_file" - # Section 1: file-path-based modules - if [ -n "$pl" ]; then - echo ":test_tube: **CI tested the following changed modules:**" >> "$comment_file" - echo "" >> "$comment_file" - for w in $(echo "$pl" | tr ',' '\n'); do - echo "- \`$w\`" >> "$comment_file" - done - - if [[ "${testedDependents}" = "false" ]]; then + # If the Scalpel report exists, show affected modules + if [ -f "$report" ]; then + # Check if full build was triggered + local full_build + full_build=$(jq -r '.fullBuildTriggered' "$report") + if [ "$full_build" = "true" ]; then + local trigger_file + trigger_file=$(jq -r '.triggerFile // "unknown"' "$report") + echo ":test_tube: **Full build triggered** — all modules tested" >> "$comment_file" echo "" >> "$comment_file" - echo "> :information_source: Dependent modules were not tested because the total number of affected modules exceeded the threshold (${maxNumberOfTestableProjects}). Use the \`test-dependents\` label to force testing all dependents." >> "$comment_file" - fi - fi + echo "Trigger: \`${trigger_file}\` matched full-build trigger pattern" >> "$comment_file" + else + # Extract affected modules by category + local direct_count upstream_count downstream_count total_count + direct_count=$(jq -r '[.affectedModules[] | select(.category == "DIRECT")] | length' "$report" 2>/dev/null || echo "0") + upstream_count=$(jq -r '[.affectedModules[] | select(.category == "UPSTREAM")] | length' "$report" 2>/dev/null || echo "0") + downstream_count=$(jq -r '[.affectedModules[] | select(.category == "DOWNSTREAM")] | length' "$report" 2>/dev/null || echo "0") + total_count=$(jq -r '.affectedModules | length' "$report" 2>/dev/null || echo "0") + + # Section 1: Directly affected modules with reasons + if [ "$direct_count" -gt 0 ]; then + echo ":test_tube: **CI tested ${direct_count} directly affected module(s):**" >> "$comment_file" + echo "" >> "$comment_file" + jq -r '.affectedModules[] | select(.category == "DIRECT") | "- `\(.path)` (\(.reasons | join(", ")))"' "$report" >> "$comment_file" + fi - # Section 2: pom dependency-detected modules - if [ -n "$dep_ids" ]; then - echo "" >> "$comment_file" - echo ":white_check_mark: **POM dependency changes: targeted tests included**" >> "$comment_file" - echo "" >> "$comment_file" - if [ -n "$changed_props_summary" ]; then - echo "Changed properties: ${changed_props_summary}" >> "$comment_file" - echo "" >> "$comment_file" - fi - if [ -n "$managed_deps_summary" ]; then - echo "Changed managed dependencies: ${managed_deps_summary}" >> "$comment_file" - echo "" >> "$comment_file" - fi - if [ -n "$managed_plugins_summary" ]; then - echo "Changed managed plugins: ${managed_plugins_summary}" >> "$comment_file" - echo "" >> "$comment_file" - fi - local dep_count - dep_count=$(echo "$dep_ids" | tr ',' '\n' | wc -l | tr -d ' ') - echo "
Modules affected by dependency changes (${dep_count})" >> "$comment_file" - echo "" >> "$comment_file" - echo "$dep_ids" | tr ',' '\n' | while read -r m; do - echo "- \`$m\`" >> "$comment_file" - done - echo "" >> "$comment_file" - echo "
" >> "$comment_file" - if [ -n "$managed_deps_summary" ] || [ -n "$managed_plugins_summary" ]; then - echo "" >> "$comment_file" - echo "> :microscope: Detected via [Maveniverse Scalpel](https://github.com/maveniverse/scalpel) effective POM comparison" >> "$comment_file" + # Section 2: Changed properties, managed deps, managed plugins + local changed_props managed_deps managed_plugins + changed_props=$(jq -r '(.changedProperties // []) | if length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true) + managed_deps=$(jq -r '(.changedManagedDependencies // []) | if length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true) + managed_plugins=$(jq -r '(.changedManagedPlugins // []) | if length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true) + + if [ -n "$changed_props" ] || [ -n "$managed_deps" ] || [ -n "$managed_plugins" ]; then + echo "" >> "$comment_file" + echo ":white_check_mark: **POM changes detected:**" >> "$comment_file" + echo "" >> "$comment_file" + [ -n "$changed_props" ] && echo "- Properties: ${changed_props}" >> "$comment_file" + [ -n "$managed_deps" ] && echo "- Managed dependencies: ${managed_deps}" >> "$comment_file" + [ -n "$managed_plugins" ] && echo "- Managed plugins: ${managed_plugins}" >> "$comment_file" + fi + + # Section 3: Downstream modules (tested as dependents) + if [ "$downstream_count" -gt 0 ]; then + echo "" >> "$comment_file" + echo "
Downstream modules also tested (${downstream_count})" >> "$comment_file" + echo "" >> "$comment_file" + jq -r '.affectedModules[] | select(.category == "DOWNSTREAM") | "- `\(.path)`"' "$report" >> "$comment_file" + echo "" >> "$comment_file" + echo "
" >> "$comment_file" + fi + + # Section 4: Upstream modules (compiled only, tests skipped) + if [ "$upstream_count" -gt 0 ]; then + echo "" >> "$comment_file" + echo "
Upstream modules compiled (tests skipped) (${upstream_count})" >> "$comment_file" + echo "" >> "$comment_file" + jq -r '.affectedModules[] | select(.category == "UPSTREAM") | "- `\(.path)`"' "$report" >> "$comment_file" + echo "" >> "$comment_file" + echo "
" >> "$comment_file" + fi + + # Empty state (no Scalpel-detected modules) + if [ "$total_count" -eq 0 ] && [ -z "$extra_modules" ]; then + echo ":information_source: No affected modules detected — no targeted tests were run." >> "$comment_file" + fi fi fi - # Section 3: extra modules (from /component-test) + # Section: Extra modules (from /component-test) — shown regardless of Scalpel report if [ -n "$extra_modules" ]; then echo "" >> "$comment_file" echo ":heavy_plus_sign: **Additional modules tested** (via \`/component-test\`):" >> "$comment_file" @@ -412,332 +184,68 @@ writeComment() { done fi - if [ -z "$pl" ] && [ -z "$dep_ids" ] && [ -z "$extra_modules" ]; then - echo ":information_source: CI did not run targeted module tests." >> "$comment_file" + # If nothing at all was shown + if [ ! -f "$report" ] && [ -z "$extra_modules" ]; then + echo ":information_source: CI did not produce a change detection report." >> "$comment_file" + fi + + # Attribution + if [ -f "$report" ]; then + echo "" >> "$comment_file" + echo "> :microscope: Detected via [Maveniverse Scalpel](https://github.com/maveniverse/scalpel) (\`skip-tests\` mode)" >> "$comment_file" fi } # ── Main ─────────────────────────────────────────────────────────────── main() { - local mavenBinary=${1} - local prId=${2} - local repository=${3} - local extraModules=${4:-} - local log="incremental-test.log" - local ret=0 - local testedDependents="" - - # Check for skip-tests label (only for PR builds) - if [ -n "$prId" ]; then - local mustSkipTests - mustSkipTests=$(hasLabel "${prId}" "skip-tests" "${repository}") - if [[ ${mustSkipTests} = "1" ]]; then - echo "The skip-tests label has been detected, no tests will be launched" - echo "" > "incremental-test-comment.md" - echo ":information_source: CI did not run targeted module tests (skip-tests label detected)." >> "incremental-test-comment.md" - exit 0 - fi - fi - - # Fetch the diff (PR diff via API, or git diff for push builds) - local diff_body - if [ -n "$prId" ]; then - echo "Fetching PR #${prId} diff..." - diff_body=$(fetchDiff "$prId" "$repository") - else - echo "No PR ID, using git diff HEAD~1..." - diff_body=$(git diff HEAD~1 2>/dev/null || true) - fi - - if [ -z "$diff_body" ]; then - echo "Could not fetch diff, skipping tests" - exit 0 - fi - - # ── Step 1: File-path analysis ── - echo "Searching for affected projects by file path..." - local projects - projects=$(echo "$diff_body" | sed -n -e '/^diff --git a/p' | awk '{print $3}' | cut -b 3- | sed 's|\(.*\)/.*|\1|' | uniq | sort) - - local pl="" - local lastProjectRoot="" - local totalAffected=0 - local pom_files="" - - for project in ${projects}; do - if [[ ${project} == */archetype-resources ]]; then - continue - elif [[ ${project} != .* ]]; then - local projectRoot - projectRoot=$(findProjectRoot "${project}") - if [[ ${projectRoot} = "." ]]; then - echo "The root project is affected, skipping targeted module testing" - echo "" > "incremental-test-comment.md" - echo ":information_source: CI did not run targeted module tests (root project files changed)." >> "incremental-test-comment.md" - exit 0 - elif [[ ${projectRoot} != "${lastProjectRoot}" ]]; then - totalAffected=$((totalAffected + 1)) - pl="$pl,${projectRoot}" - lastProjectRoot=${projectRoot} - fi - fi - done - pl="${pl:1}" # strip leading comma - - # Only analyze parent/pom.xml for grep-based dependency detection - # (matches original detect-test.sh behavior) - if echo "$diff_body" | grep -q '^diff --git a/parent/pom.xml'; then - pom_files="parent/pom.xml" - fi - - # ── Step 2: POM dependency analysis (dual: grep + Scalpel) ── - # Variables shared with analyzePomDependencies/findAffectedModules - local dep_module_ids="" - local all_changed_props="" - # Scalpel results (not local — set by runScalpelDetection) - scalpel_module_ids="" - scalpel_module_paths="" - scalpel_props="" - scalpel_managed_deps="" - scalpel_managed_plugins="" - - # Step 2a: Grep-based detection (existing approach) - if [ -n "$pom_files" ]; then - echo "" - echo "Analyzing parent POM dependency changes (grep)..." - while read -r pom_file; do - [ -z "$pom_file" ] && continue - - # Capture changed props for this pom before calling analyze - local pom_diff - pom_diff=$(extractPomDiff "$diff_body" "$pom_file") - if [ -n "$pom_diff" ]; then - local props - props=$(detectChangedProperties "$pom_diff") - if [ -n "$props" ]; then - all_changed_props="${all_changed_props:+${all_changed_props}, }$(echo "$props" | tr '\n' ',' | sed 's/,$//')" - fi - fi - - analyzePomDependencies "$diff_body" "$pom_file" - done <<< "$pom_files" - fi - - # Step 2b: Scalpel detection (parallel, for any pom.xml change) - # Scalpel uses effective POM model comparison — catches managed deps, - # plugin changes, and transitive impacts that grep misses. - if echo "$diff_body" | grep -q '^diff --git a/.*pom\.xml'; then - echo "" - echo "Running Scalpel POM analysis..." - runScalpelDetection - fi + local extraModules="${1:-}" + local report="target/scalpel-report.json" + local comment_file="incremental-test-comment.md" + local build_failed="${BUILD_FAILED:-false}" - # Step 2c: Merge grep and Scalpel results (union, deduplicated) - if [ -n "$scalpel_module_ids" ]; then - dep_module_ids="${dep_module_ids:+${dep_module_ids},}${scalpel_module_ids}" - dep_module_ids=$(echo "$dep_module_ids" | tr ',' '\n' | sort -u | tr '\n' ',' | sed 's/,$//') - fi - if [ -n "$scalpel_props" ]; then - if [ -z "$all_changed_props" ]; then - all_changed_props="$scalpel_props" - else - # Merge and deduplicate property names - all_changed_props=$(printf '%s, %s' "$all_changed_props" "$scalpel_props" | tr ',' '\n' | sed 's/^ *//' | sort -u | tr '\n' ',' | sed 's/,$//' | sed 's/,/, /g') - fi - fi + echo "Generating test report..." - # ── Step 3: Merge and deduplicate ── - # Separate file-path modules into testable (has src/test) and pom-only. - # Pom-only modules (e.g. "parent") are kept in the build list but must NOT - # be expanded with -amd, since that would pull in every dependent module. - local testable_pl="" - local pom_only_pl="" - for w in $(echo "$pl" | tr ',' '\n'); do - if [ -d "$w/src/test" ]; then - testable_pl="${testable_pl:+${testable_pl},}${w}" - else - pom_only_pl="${pom_only_pl:+${pom_only_pl},}${w}" - echo " Pom-only module (no src/test, won't expand dependents): $w" - fi - done + # Generate PR comment from Scalpel report + writeComment "$comment_file" "$report" "$extraModules" - # Build final_pl: testable file-path modules + dependency-detected + pom-only + extra - local final_pl="" - if [ -n "$testable_pl" ]; then - final_pl="$testable_pl" + # Get tested module paths (DIRECT + DOWNSTREAM) for disabled-test and IT scanning + local tested_paths="" + if [ -f "$report" ]; then + tested_paths=$(jq -r '.affectedModules[] | select(.category == "DIRECT" or .category == "DOWNSTREAM") | .path' "$report" 2>/dev/null || true) fi - if [ -n "$dep_module_ids" ]; then - final_pl="${final_pl:+${final_pl},}${dep_module_ids}" - fi - if [ -n "$pom_only_pl" ]; then - final_pl="${final_pl:+${final_pl},}${pom_only_pl}" - fi - - # Merge extra modules (e.g. from /component-test) + # Include extra modules in the scan if [ -n "$extraModules" ]; then - echo "" - echo "Extra modules requested: $extraModules" - final_pl="${final_pl:+${final_pl},}${extraModules}" - fi - - if [ -z "$final_pl" ]; then - echo "" - echo "No modules to test" - writeComment "incremental-test-comment.md" "" "" "" "" "" "" "" - exit 0 - fi - - echo "" - echo "Modules to test:" - for w in $(echo "$final_pl" | tr ',' '\n'); do - echo " - $w" - done - echo "" - - # ── Step 4: Run tests ── - # Decide whether to use -amd (also-make-dependents): - # - Use -amd when there are testable file-path modules (to test their dependents) - # - Subject to threshold check to avoid testing too many modules - # - Pom-only modules are excluded from -pl to prevent -amd from pulling in everything - # (Maven builds them implicitly as dependencies of child modules) - local use_amd=false - local testDependents="0" - - if [ -n "$testable_pl" ]; then - # File-path modules with tests — use -amd to catch dependents - if [ -n "$prId" ]; then - testDependents=$(hasLabel "${prId}" "test-dependents" "${repository}") - fi - - if [[ ${testDependents} = "1" ]]; then - echo "The test-dependents label has been detected, testing dependents too" - use_amd=true - testedDependents=true - else - # Include extra modules in the count — with -amd, Maven expands all of them - local threshold_pl="$testable_pl" - if [ -n "$extraModules" ]; then - threshold_pl="${threshold_pl},${extraModules}" - fi - local totalTestableProjects - totalTestableProjects=$(./mvnw -B -q -amd exec:exec -Dexec.executable="pwd" -pl "$threshold_pl" 2>/dev/null | wc -l) || true - totalTestableProjects=$(echo "$totalTestableProjects" | tail -1 | tr -d '[:space:]') - totalTestableProjects=${totalTestableProjects:-0} - - if [[ ${totalTestableProjects} -gt ${maxNumberOfTestableProjects} ]]; then - echo "Too many dependent modules (${totalTestableProjects} > ${maxNumberOfTestableProjects}), testing only the affected modules" - testedDependents=false - else - echo "Testing affected modules and their dependents (${totalTestableProjects} modules)" - use_amd=true - testedDependents=true - fi - fi - elif [ -n "$dep_module_ids" ]; then - # Only dependency-detected modules (no file-path code changes) - echo "POM dependency analysis found affected modules — testing specific modules" - testedDependents=true - else - # Only pom-only modules, no testable code and no dependency results - echo "Only pom-only modules changed with no detected dependency impact" - testedDependents=true + tested_paths="$tested_paths $(echo "$extraModules" | tr ',' ' ')" fi - # Build the -pl argument: - # - Exclude pom-only modules from -pl when using -amd (they'd pull in everything) - # - Append exclusion list when dependency-detected modules are present - local build_pl="$final_pl" - if [[ "$use_amd" = true ]] && [ -n "$pom_only_pl" ]; then - # Remove pom-only modules — Maven builds them implicitly as dependencies - build_pl="" - if [ -n "$testable_pl" ]; then - build_pl="$testable_pl" - fi - if [ -n "$dep_module_ids" ]; then - build_pl="${build_pl:+${build_pl},}${dep_module_ids}" - fi - if [ -n "$extraModules" ]; then - build_pl="${build_pl:+${build_pl},}${extraModules}" - fi - fi - # This needs to install, not just test, otherwise test-infra will fail due to jandex maven plugin - # Exclusion list is only needed with -amd (to prevent testing generated/meta modules); - # without -amd, only the explicitly listed modules are built. - if [[ "$use_amd" = true ]]; then - $mavenBinary -l "$log" $MVND_OPTS install -pl "${build_pl},${EXCLUSION_LIST}" -amd || ret=$? - else - $mavenBinary -l "$log" $MVND_OPTS install -pl "$build_pl" || ret=$? - fi - - # ── Step 5: Write comment and summary ── - local comment_file="incremental-test-comment.md" - writeComment "$comment_file" "$pl" "$dep_module_ids" "$all_changed_props" "$testedDependents" "$extraModules" "$scalpel_managed_deps" "$scalpel_managed_plugins" - - # Check for tests disabled in CI via @DisabledIfSystemProperty(named = "ci.env.name") - local disabled_tests - disabled_tests=$(detectDisabledTests "$final_pl") - if [ -n "$disabled_tests" ]; then - echo "" >> "$comment_file" - echo ":warning: **Some tests are disabled on GitHub Actions** (\`@DisabledIfSystemProperty(named = \"ci.env.name\")\`) and require manual verification:" >> "$comment_file" - echo "$disabled_tests" >> "$comment_file" - fi - - # Check for excluded IT suites that should be run manually - checkManualItTests "$final_pl" "$comment_file" - - # Append reactor module list from build log - if [[ -f "$log" ]]; then - local reactor_modules - reactor_modules=$(grep '^\[INFO\] Camel ::' "$log" | sed 's/\[INFO\] //' | sed 's/ \..*$//' | sed 's/ *\[.*\]$//' | sort -u || true) - if [[ -n "$reactor_modules" ]]; then - local count - count=$(echo "$reactor_modules" | wc -l | tr -d ' ') - local reactor_label - if [[ "${testedDependents}" = "false" ]]; then - reactor_label="Build reactor — dependencies compiled but only changed modules were tested" - else - reactor_label="All tested modules" - fi - + # Check for tests disabled in CI + if [ -n "$tested_paths" ]; then + local disabled_tests + disabled_tests=$(detectDisabledTests $tested_paths) + if [ -n "$disabled_tests" ]; then echo "" >> "$comment_file" - echo "
${reactor_label} ($count modules)" >> "$comment_file" - echo "" >> "$comment_file" - - if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then - echo "" >> "$GITHUB_STEP_SUMMARY" - echo "
${reactor_label} ($count)" >> "$GITHUB_STEP_SUMMARY" - echo "" >> "$GITHUB_STEP_SUMMARY" - fi - - echo "$reactor_modules" | while read -r m; do - [ -n "${GITHUB_STEP_SUMMARY:-}" ] && echo "- $m" >> "$GITHUB_STEP_SUMMARY" - echo "- $m" >> "$comment_file" - done - - if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then - echo "" >> "$GITHUB_STEP_SUMMARY" - echo "
" >> "$GITHUB_STEP_SUMMARY" - fi - echo "" >> "$comment_file" - echo "
" >> "$comment_file" + echo ":warning: **Some tests are disabled on GitHub Actions** (\`@DisabledIfSystemProperty(named = \"ci.env.name\")\`) and require manual verification:" >> "$comment_file" + echo "$disabled_tests" >> "$comment_file" fi + + # Check for excluded IT suites that should be run manually + checkManualItTests "$comment_file" $tested_paths fi - # Write step summary header - if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then + # Write step summary + if [ -n "${GITHUB_STEP_SUMMARY:-}" ] && [ -f "$report" ]; then { echo "" echo "### Tested modules" echo "" - for w in $(echo "$final_pl" | tr ',' '\n'); do - echo "- \`$w\`" - done + jq -r '.affectedModules[] | select(.category == "DIRECT" or .category == "DOWNSTREAM") | "- `\(.path)` [\(.category)] \(.reasons | join(", "))"' "$report" 2>/dev/null || true echo "" } >> "$GITHUB_STEP_SUMMARY" fi - if [[ ${ret} -ne 0 ]]; then + # Parse test failures from surefire/failsafe reports + if [ "$build_failed" = "true" ]; then echo "Processing surefire and failsafe reports to create the summary" if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then echo -e "| Failed Test | Duration | Failure Type |\n| --- | --- | --- |" >> "$GITHUB_STEP_SUMMARY" @@ -745,7 +253,7 @@ main() { find . -path '*target/*-reports*' -iname '*.txt' -exec .github/actions/incremental-build/parse_errors.sh {} \; fi - exit $ret + echo "Test report generated: $comment_file" } main "$@" diff --git a/.github/workflows/pr-build-main.yml b/.github/workflows/pr-build-main.yml index 9588a33070821..02b223cdc7658 100644 --- a/.github/workflows/pr-build-main.yml +++ b/.github/workflows/pr-build-main.yml @@ -88,9 +88,31 @@ jobs: distribution: 'temurin' java-version: ${{ matrix.java }} cache: 'maven' - - name: maven build + - name: Check skip-tests label + id: labels + if: ${{ github.event.number || inputs.pr_number }} + shell: bash + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + PR_NUMBER: ${{ github.event.number || inputs.pr_number }} + run: | + has_skip=$(curl -s \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "https://api.github.com/repos/${{ github.repository }}/issues/${PR_NUMBER}/labels" \ + | jq -r '.[].name' | { grep -c "incremental-skip-tests" || true; }) + echo "skip-tests=${has_skip}" >> $GITHUB_OUTPUT + - name: Build and test + id: build-and-test if: ${{ !inputs.skip_full_build }} - run: ./etc/scripts/regen.sh + run: | + if [ "${{ steps.labels.outputs.skip-tests }}" = "1" ]; then + echo "The skip-tests label has been detected, skipping tests" + ./etc/scripts/regen.sh --skip-tests + else + ./etc/scripts/regen.sh + fi - name: Quick dependency build if: ${{ inputs.skip_full_build }} shell: bash @@ -104,23 +126,24 @@ jobs: name: build-${{ matrix.java }}.log path: build.log - name: Fail if there are uncommitted changes - if: ${{ !inputs.skip_full_build }} + if: always() && !inputs.skip_full_build && steps.build-and-test.outcome != 'skipped' shell: bash run: | [[ -z $(git status --porcelain) ]] || { echo 'There are uncommitted changes'; git status; echo; echo; git diff; exit 1; } - - name: mvn test + - name: Test extra modules + id: test-extra + if: ${{ inputs.skip_full_build && inputs.extra_modules }} + shell: bash + env: + EXTRA_MODULES: ${{ inputs.extra_modules }} + run: | + ${{ steps.install-mvnd.outputs.mvnd-dir }}/mvnd -l build.log $MVND_OPTS install -pl "$EXTRA_MODULES" || true + - name: Generate test report + if: always() uses: ./.github/actions/incremental-build with: - pr-id: ${{ github.event.number || inputs.pr_number }} - github-token: ${{ secrets.GITHUB_TOKEN }} - skip-mvnd-install: 'true' extra-modules: ${{ inputs.extra_modules || '' }} - - name: archive incremental test logs - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 - if: always() - with: - name: incremental-test-java-${{ matrix.java }}.log - path: incremental-test.log + build-failed: ${{ steps.build-and-test.outcome == 'failure' || steps.test-extra.outcome == 'failure' }} # All non-experimental JDK matrix entries upload with overwrite: true. # This ensures a comment is posted even if one JDK build fails — the # content is identical across JDKs (same modules tested), so last writer wins. diff --git a/.github/workflows/sonar-build.yml b/.github/workflows/sonar-build.yml index 9b1e990af7213..decd1b1680899 100644 --- a/.github/workflows/sonar-build.yml +++ b/.github/workflows/sonar-build.yml @@ -56,20 +56,11 @@ jobs: java-version: '21' cache: 'maven' - - name: Build with Maven - run: ./mvnw install -B -Dquickly - - # Run core tests with JaCoCo and generate aggregated coverage report. - # The source modules must be in the reactor so report-aggregate can - # map execution data from camel-core tests to their classes. - # TODO: Once incremental-build.sh supports module detection in the sonar - # workflow, replace the hardcoded -pl with detected affected modules - # to get coverage on components too (see PR #22247). - - name: Run tests with coverage and generate report - run: > - ./mvnw verify -B -Dcoverage - -pl core/camel-api,core/camel-util,core/camel-support,core/camel-management-api,core/camel-management,core/camel-base,core/camel-base-engine,core/camel-core-engine,core/camel-core-languages,core/camel-core-model,core/camel-core-processor,core/camel-core-reifier,core/camel-core,coverage - -Dmaven.test.failure.ignore=true + # Single invocation: build everything, test only affected modules with + # JaCoCo coverage via Scalpel skip-tests mode. Coverage reports are + # generated per-module for all tested modules (not just core). + - name: Build and test with coverage + run: ./etc/scripts/regen.sh --coverage - name: Prepare compiled classes artifact shell: bash diff --git a/.mvn/extensions.xml b/.mvn/extensions.xml index 9a8476957eb2f..0747a32c967a1 100644 --- a/.mvn/extensions.xml +++ b/.mvn/extensions.xml @@ -22,7 +22,7 @@ eu.maveniverse.maven.scalpel - extension3 - 0.1.0 + extension + 0.2.0 diff --git a/etc/scripts/regen.sh b/etc/scripts/regen.sh index 57eb8aab83657..ca02eac16a955 100755 --- a/etc/scripts/regen.sh +++ b/etc/scripts/regen.sh @@ -21,24 +21,49 @@ set -e # Move to top directory cd `dirname "$0"`/../.. +# Parse arguments +skip_tests="" +coverage="" +for arg in "$@"; do + case "$arg" in + --skip-tests) skip_tests=true ;; + --coverage) coverage=true ;; + esac +done + # Force clean git clean -fdx rm -Rf **/src/generated/ -# Regenerate everything -if ./mvnw --batch-mode -Pregen -DskipTests install >> build.log 2>&1; then - echo "✅ mvn -Pregen succeeded." +# Build flags: Scalpel skip-tests mode in CI +# On developer machines, Scalpel auto-disables (no GITHUB_BASE_REF). +scalpel_flags="" +if [ "${GITHUB_ACTIONS:-}" = "true" ] && [ -z "$skip_tests" ]; then + scalpel_flags="-Dscalpel.mode=skip-tests -Dscalpel.skipTestsForUpstream=true -Dscalpel.fetchBaseBranch=true -Dscalpel.fullBuildTriggers= -Dscalpel.reportFile=target/scalpel-report.json -Dscalpel.impactedLog=target/scalpel-impacted.txt" +fi + +extra_flags="" +if [ -n "$skip_tests" ]; then + extra_flags="-DskipTests" +fi +if [ -n "$coverage" ]; then + extra_flags="$extra_flags -Dcoverage" +fi + +# Regenerate everything (and run tests on affected modules via Scalpel in CI) +if ./mvnw --batch-mode -Pregen $scalpel_flags $extra_flags install >> build.log 2>&1; then + echo "mvn -Pregen succeeded." else - echo "❌ mvn -Pregen failed. Last 50 lines of build.log:" + echo "mvn -Pregen failed. Last 50 lines of build.log:" tail -n 50 build.log exit 1 fi # One additional pass to get the info for the 'others' jars if ./mvnw --batch-mode install -f catalog/camel-catalog >> build.log 2>&1; then - echo "✅ mvn install for camel-catalog succeeded." + echo "mvn install for camel-catalog succeeded." else - echo "❌ mvn install for camel-catalog failed. Last 50 lines of build.log:" + echo "mvn install for camel-catalog failed. Last 50 lines of build.log:" tail -n 50 build.log exit 1 fi From 1bee76d42482186e54c081c60c3132f97ebdf2a9 Mon Sep 17 00:00:00 2001 From: Guillaume Nodet Date: Fri, 10 Apr 2026 01:16:43 +0200 Subject: [PATCH 2/4] ci: fix Scalpel fetch failure in shallow CI clones Pre-fetch the base branch using git CLI in the workflow instead of relying on Scalpel's JGit fetchBaseBranch=true. JGit's FetchCommand crashes with MissingObjectException during fetchSubmodules in GitHub Actions' shallow clones, even with fetchBaseBranch enabled. - Remove persist-credentials: false so git fetch works after checkout - Add "Fetch base branch" step using git CLI (handles shallow clones) - Set scalpel.fetchBaseBranch=false (base branch already available) Co-Authored-By: Claude Opus 4.6 --- .github/CI-ARCHITECTURE.md | 2 +- .github/workflows/pr-build-main.yml | 7 ++++++- .github/workflows/sonar-build.yml | 6 +++--- etc/scripts/regen.sh | 2 +- 4 files changed, 11 insertions(+), 6 deletions(-) diff --git a/.github/CI-ARCHITECTURE.md b/.github/CI-ARCHITECTURE.md index c6ade2526f912..871b39a31d5ee 100644 --- a/.github/CI-ARCHITECTURE.md +++ b/.github/CI-ARCHITECTURE.md @@ -131,7 +131,7 @@ Scalpel flags are set in `etc/scripts/regen.sh` when running in CI (`GITHUB_ACTI | --- | --- | --- | | `scalpel.mode` | `skip-tests` | Build all, test only affected | | `scalpel.skipTestsForUpstream` | `true` | Don't test upstream-only modules | -| `scalpel.fetchBaseBranch` | `true` | Auto-fetch base branch in shallow clones | +| `scalpel.fetchBaseBranch` | `false` | Base branch pre-fetched by workflow (git CLI handles shallow clones better than JGit) | | `scalpel.fullBuildTriggers` | *(empty)* | Override `.mvn/**` default | | `scalpel.reportFile` | `target/scalpel-report.json` | JSON report for PR comment | | `scalpel.impactedLog` | `target/scalpel-impacted.txt` | Simple module path list | diff --git a/.github/workflows/pr-build-main.yml b/.github/workflows/pr-build-main.yml index 02b223cdc7658..685ee5118ed84 100644 --- a/.github/workflows/pr-build-main.yml +++ b/.github/workflows/pr-build-main.yml @@ -76,8 +76,13 @@ jobs: steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: - persist-credentials: false ref: ${{ inputs.pr_ref || '' }} + - name: Fetch base branch for Scalpel change detection + if: ${{ !inputs.skip_full_build }} + run: | + # Scalpel needs the base branch to detect changes. + # git CLI handles shallow clones reliably (JGit's fetchSubmodules does not). + git fetch --no-tags --depth=200 origin "${GITHUB_BASE_REF:-main}:refs/remotes/origin/${GITHUB_BASE_REF:-main}" - id: install-packages uses: ./.github/actions/install-packages - id: install-mvnd diff --git a/.github/workflows/sonar-build.yml b/.github/workflows/sonar-build.yml index decd1b1680899..58b5c5274baf9 100644 --- a/.github/workflows/sonar-build.yml +++ b/.github/workflows/sonar-build.yml @@ -43,9 +43,9 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - with: - persist-credentials: false - + - name: Fetch base branch for Scalpel change detection + run: | + git fetch --no-tags --depth=200 origin "${GITHUB_BASE_REF:-main}:refs/remotes/origin/${GITHUB_BASE_REF:-main}" - id: install-packages uses: ./.github/actions/install-packages diff --git a/etc/scripts/regen.sh b/etc/scripts/regen.sh index ca02eac16a955..46ee6f52341fb 100755 --- a/etc/scripts/regen.sh +++ b/etc/scripts/regen.sh @@ -39,7 +39,7 @@ rm -Rf **/src/generated/ # On developer machines, Scalpel auto-disables (no GITHUB_BASE_REF). scalpel_flags="" if [ "${GITHUB_ACTIONS:-}" = "true" ] && [ -z "$skip_tests" ]; then - scalpel_flags="-Dscalpel.mode=skip-tests -Dscalpel.skipTestsForUpstream=true -Dscalpel.fetchBaseBranch=true -Dscalpel.fullBuildTriggers= -Dscalpel.reportFile=target/scalpel-report.json -Dscalpel.impactedLog=target/scalpel-impacted.txt" + scalpel_flags="-Dscalpel.mode=skip-tests -Dscalpel.skipTestsForUpstream=true -Dscalpel.fetchBaseBranch=false -Dscalpel.fullBuildTriggers= -Dscalpel.reportFile=target/scalpel-report.json -Dscalpel.impactedLog=target/scalpel-impacted.txt" fi extra_flags="" From b1e46e07ca4cf5ec148fdbb7ca02d70bc1f0acc5 Mon Sep 17 00:00:00 2001 From: Guillaume Nodet Date: Fri, 10 Apr 2026 01:18:00 +0200 Subject: [PATCH 3/4] ci: temporarily allow fork CI runs for testing Co-Authored-By: Claude Opus 4.6 --- .github/workflows/pr-build-main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/pr-build-main.yml b/.github/workflows/pr-build-main.yml index 685ee5118ed84..5d79c2fb6e228 100644 --- a/.github/workflows/pr-build-main.yml +++ b/.github/workflows/pr-build-main.yml @@ -60,7 +60,7 @@ permissions: jobs: build: - if: github.repository == 'apache/camel' + if: github.repository == 'apache/camel' || github.repository == 'gnodet/camel' permissions: contents: read runs-on: ubuntu-latest From 1347cea566e8f139ea6e12c482c59e9cfe72ad48 Mon Sep 17 00:00:00 2001 From: Guillaume Nodet Date: Fri, 10 Apr 2026 01:32:44 +0200 Subject: [PATCH 4/4] ci: deepen PR branch history for Scalpel merge-base detection The checkout at depth=1 means the merge base between the PR and the base branch is unreachable. Add git fetch --deepen=200 to make the PR branch history available alongside the base branch. Co-Authored-By: Claude Opus 4.6 --- .github/workflows/pr-build-main.yml | 5 +++-- .github/workflows/sonar-build.yml | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pr-build-main.yml b/.github/workflows/pr-build-main.yml index 5d79c2fb6e228..443eae4134b2e 100644 --- a/.github/workflows/pr-build-main.yml +++ b/.github/workflows/pr-build-main.yml @@ -80,8 +80,9 @@ jobs: - name: Fetch base branch for Scalpel change detection if: ${{ !inputs.skip_full_build }} run: | - # Scalpel needs the base branch to detect changes. - # git CLI handles shallow clones reliably (JGit's fetchSubmodules does not). + # Scalpel needs the merge base between HEAD and the base branch. + # The checkout is depth=1, so deepen both sides for merge-base reachability. + git fetch --deepen=200 2>/dev/null || true git fetch --no-tags --depth=200 origin "${GITHUB_BASE_REF:-main}:refs/remotes/origin/${GITHUB_BASE_REF:-main}" - id: install-packages uses: ./.github/actions/install-packages diff --git a/.github/workflows/sonar-build.yml b/.github/workflows/sonar-build.yml index 58b5c5274baf9..58ffe26d7e0c2 100644 --- a/.github/workflows/sonar-build.yml +++ b/.github/workflows/sonar-build.yml @@ -45,6 +45,7 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - name: Fetch base branch for Scalpel change detection run: | + git fetch --deepen=200 2>/dev/null || true git fetch --no-tags --depth=200 origin "${GITHUB_BASE_REF:-main}:refs/remotes/origin/${GITHUB_BASE_REF:-main}" - id: install-packages uses: ./.github/actions/install-packages