foreman-packaging: Bootstrap foreman-build and tiered EL10 build execution

[zjhuntin]

Summary

Execute the full EL10 build in dependency order using a deliberate, PR-based approach. Instead of bulk build triggers, create PRs to foreman-packaging for each batch of packages, keeping ~10 PRs open at a time.

Use scripts/el10_rebuild_tracker.py to manage the pipeline.

Prerequisites

• foreman-packaging: EL10 server buildroot infrastructure #2 (buildroot infrastructure / Add EL10 buildroot support for foreman, plugins, and katello Copr projects foreman-packaging#13755)
• foreman-packaging: RHEL-137712 nodejs path workaround #3 (RHEL-137712 workaround)
• foreman-packaging: Regenerate all nodejs packages with upstream npm2rpm #4 (nodejs regeneration)

Approach

For each phase in the build order, attempt scratch builds for batches of ~10 packages. Fix any failures, then create PRs to theforeman/foreman-packaging (branch: rpm/develop) with the spec fixes. Keep ~10 PRs in flight to balance CI load.

Workflow per batch:

1. ./scripts/el10_rebuild_tracker.py next — see next packages to attempt
2. cd workspace/foreman-packaging && obal scratch <packages> --scm none — attempt builds
3. Fix any spec failures (library paths, Ruby 3.3 compat, macro changes, etc.)
4. Create a PR to theforeman/foreman-packaging with the fixes
5. ./scripts/el10_rebuild_tracker.py pr foreman-packaging#NNNN <packages> — record the PR
6. After merge: ./scripts/el10_rebuild_tracker.py merged foreman-packaging#NNNN

Build Phases (in dependency order)

Phase 0: Bootstrap
  foreman-build subpackage (one-time, manual bootstrap)

Phase 1: ruby-tier1 (176 packages)
Phase 2: ruby-tier2 (4 packages)
Phase 3: frameworks — rails + hammer + other (19 packages, parallel)
Phase 4: nodejs (118 packages)
Phase 5: core-apps — foreman + installer + proxy (26 packages)
Phase 6: plugins-tier1 (93 packages)
Phase 7: plugins-tier2 (110 packages)
Phase 8: katello (32 packages)

Notes

• Installer ordering: foreman-installer is in the same manifest group as its build deps (rubygem-kafo, etc.). Build the deps first, then foreman-installer. Two prior PRs to fix this at the manifest level were closed without merging (#8814, #9564).
• rubygem-grpc: Fails on both EL9 and EL10. Used by foreman_google plugin, not on the critical path.
• Many packages will build fine without changes — mark them as ok in the tracker and only create PRs for packages that need spec fixes.
• ekohl's Copr at https://copr.fedorainfracloud.org/coprs/ekohl/foreman-nightly-staging/ has successful EL10 builds of foreman core — confirms the path works.

[zjhuntin]

Progress Update — 2026-07-24

Completed

• Disabled EL10 repoclosure in all three layers:
  • foreman-packaging package_manifest.yaml (PR Temporarily disable EL10 repoclosure during rebuild foreman-packaging#13771, merged)
  • pulpcore-packaging package_manifest.yaml (PR Temporarily disable EL10 repoclosure during rebuild pulpcore-packaging#2788, merged)
  • Jenkins pipeline rpm_copr_packaging.groovy — added skip_repoclosure_dists = ['el10'] (PR Skip EL10 repoclosure during rebuild jenkins-jobs#591, merged)
• Created automation scripts: el10_rebuild_tracker.py and el10_create_batch_pr.py (PR Add EL10 rebuild plan, analysis, and tracker script #21)
• Opened first 10 ruby-tier1 rebuild PRs: Rebuild libsass for EL10 foreman-packaging#13761–#13770

Blocker: Copr copr_base repository timeout

All 10 PRs are failing CI with: "Giving up waiting for copr_base repository"

The Copr backend creates a new scratch project per PR, then waits for repomd.xml to appear. It polls every 2 seconds for ~60 seconds, then fails. This affects both EL9 and EL10 chroots on the scratch project. copr-cli regenerate-repos was attempted but doesn't fix already-failed builds. This appears to be a Copr infrastructure issue, not specific to our workload.

Next steps

• Investigate Copr infrastructure issue (may need to file upstream at pagure.io/copr/copr)
• Once builds pass, continue with remaining 166 ruby-tier1 packages

[ekohl]

Execute the full EL10 build in dependency order using a deliberate, PR-based approach. Instead of bulk build triggers, create PRs to foreman-packaging for each batch of packages, keeping ~10 PRs open at a time.

Use scripts/el10_rebuild_tracker.py to manage the pipeline.

In my experience manually kicking off all builds in the real repository instead of opening PRs is a lot faster.

obal can report all the job IDs. You can then monitor them with copr watch-build, which will poll them for you. Tracking down any failures from that is probably easier than using PRs.

The Copr backend creates a new scratch project per PR, then waits for repomd.xml to appear. It polls every 2 seconds for ~60 seconds, then fails. This affects both EL9 and EL10 chroots on the scratch project. copr-cli regenerate-repos was attempted but doesn't fix already-failed builds. This appears to be a Copr infrastructure issue, not specific to our workload.

This is because our build infra for PRs has a lot of overhead.

[zjhuntin]

Progress update (2026-07-28):

Phase 1: ruby-tier1 — COMPLETE

All 176 packages merged across PRs #13761–#13794. Final 5 interdep packages (dynflow, powerbar, rack-jsonp, sass-rails, sassc-rails) in PR #13822.

Phase 6: ruby_plugin_packages — IN PROGRESS

70 packages across 7 PRs:

• #13815: acts_as_list..bcrypt_pbkdf (10)
• #13816: chunky_png..dry-initializer (10)
• #13817: dry-logic..fog-kubevirt (10)
• #13818: fog-ovirt..google-cloud-errors (10)
• #13819: google-protobuf..jsonpath (10)
• #13820: kubeclient..raabro (10)
• #13821: rainbow..grpc (10)

Phase 4: nodejs — BLOCKED

Waiting on theforeman/npm2rpm#97 merge + release. Tooling PR #13813 is ready.

Overall

• 176/578 packages done (30.4%)
• 75 packages in open PRs
• All non-nodejs groups (253 packages) have deps satisfied and can proceed
• nodejs group (122 packages) blocked on npm2rpm#97

[zjhuntin]

Progress Update (2026-07-28)

Tier 1 (176 packages): All merged. PRs #13761–#13794 (170 independent) + #13822 (5 interdep: dynflow, powerbar, rack-jsonp, sass-rails, sassc-rails).

ruby_plugin_packages + tier2 (70 packages): All merged across 7 PRs:

• #13815–#13819 (batches 1-5, 50 packages)
• #13824–#13825 (batches 6-7, 19 packages — redone with per-package commits)

rubygem-grpc: PR #13823 open, blocked on rubygem-google-protobuf + rubygem-googleapis-common-protos-types (in unmerged PR #13819). Not on critical path.

nodejs (122 packages): Still blocked on npm2rpm PR #97. Tooling PR #13813 ready.

Tooling updates:

• el10_create_batch_pr.py now creates one commit per package (easier revert/cherry-pick)
• el10_remove_from_pr.py added for removing packages from batch PRs (with --requeue support)

Next groups (all deps satisfied, can proceed):

• ruby_core_tier2, foreman_plugin_packages, hammer_plugin_packages, foreman_proxy_plugin_packages, foreman_proxy_packages, foreman_client_packages, katello packages, frameworks, core-apps

[zjhuntin]

Progress update (2026-07-29)

267/578 packages done (46.2%)

Completed phases

• ruby-tier1 (176/176): ALL MERGED
• ruby-tier2 (3/4): puma, sassc, websocket-driver merged. rkerberos pending (unmaintained upstream).
• frameworks (19/19): ALL MERGED (PRs #13848, #13849)

In progress

• core-apps: PR #13853 (24 packages) awaiting CI/merge. foreman-installer held for separate PR (depends on kafo + puppet-agent-puppet-strings).
• plugins-tier1: 69/93 ok. PR #13854 (18 hammer plugins) open. 5 packages pending (newt/fdi-service/foreman_remote_execution/yggdrasil/nodejs-react-json-tree).
• plugins-tier2: PR #13855 (20 proxy plugins + foreman-tasks) open. 91 pending.

Blocked

• nodejs (118 packages): Blocked on npm2rpm PR #97.
• katello (32 packages): Waiting on plugins-tier1/tier2.

Open PRs

• #13853: core-apps (24 packages)
• #13854: hammer plugins (18 packages)
• #13855: proxy plugins + foreman-tasks (20 packages)
• #13823: rubygem-grpc (GCC 14 pointer-type fixes done, now hitting __wrap_memcpy linker issue in %check)

[ekohl]

• rkerberos pending (unmaintained upstream).

This is no longer true. Behind the scenes I've pushed for it and https://github.com/rkerberos/rkerberos is now its own org. There was a new release that does compile on EL10.

[zjhuntin]

Progress update (2026-07-30)

363/578 packages merged (62.8%), 112 nodejs in PR, ~103 remaining

Completed phases

• ruby-tier1 (176/176): ALL MERGED
• ruby-tier2 (3/4): puma, sassc, websocket-driver merged. rkerberos has a new release that compiles on EL10 (per ekohl) — needs version bump PR.
• frameworks (19/19): ALL MERGED
• core-apps (24/26): PR #13853 merged (24 packages). foreman-installer held for kafo/puppet-strings deps.
• plugins-tier1 (87/93): All 8 PRs merged. 6 remaining: newt, fdi-service, foreman_remote_execution, yggdrasil, radcli, react-json-tree (last one done via nodejs regen).
• plugins-tier2 (44/110): PRs #13855, #13858, #13859 all merged (44 packages). 66 remaining.

In progress

• nodejs (10/122 merged, 112 in PR):
  • Batch 8 (10 packages) merged via PR #13871
  • Batches 1-6 (94 packages) in PRs #13869, #13864, #13870, #13865, #13866, #13867
  • Strategy fixes (18 packages) in PR #13872
  • All 7 PRs have CI running

Open non-nodejs PRs

• #13823: rubygem-grpc — CI passes but __wrap_memcpy linker issue in %check
• #13857: rubygem-foreman-tasks — CI fail
• #13862: python-jinja2 — CI fail
• #13863: 10 python packages — CI fail
• #13813: nodejs tooling (add_npm_package.sh) — CI fail

Remaining work (~103 packages)

• plugins-tier2: ~66 packages
• katello: ~30 packages (waiting on plugin deps)
• rkerberos: version bump needed (new upstream release available)
• grpc: linker issue
• foreman-installer, newt, radcli, fdi-service, foreman_remote_execution, yggdrasil

[zjhuntin]

@ekohl thank you for the update on that!

[zjhuntin]

Progress update (2026-07-30, evening)

428/578 packages merged (74.0%), 48 nodejs in PR, ~102 remaining

Changes since last update

• rubygem-grpc: PR #13823 MERGED. GCC 14 pointer-type + __wrap_memcpy linker issues resolved.
• nodejs batches 1–4: PRs #13869, #13864, #13870, #13865 all MERGED (64 packages).
• ruby-tier2: Now 4/4 complete (grpc was the last one).

Copr release status

71 nodejs packages released to nightly-staging today (from merged batches 1-4 + 8). 48 nodejs packages still in open PRs (batches 5, 6, strategy fixes) — not yet released. rubygem-grpc release pending next nightly run.

Completed phases

• ruby-tier1 (176/176): ALL MERGED
• ruby-tier2 (4/4): ALL MERGED (puma, sassc, websocket-driver, grpc)
• frameworks (19/19): ALL MERGED

In progress

• core-apps (24/26): 24 merged. foreman-installer held for kafo/puppet-strings deps. 1 other pending.
• plugins-tier1 (87/93): 6 remaining (newt, fdi-service, foreman_remote_execution, yggdrasil, radcli, react-json-tree).
• plugins-tier2 (44/110): 66 remaining.
• nodejs (74/122 merged, 48 in PR):
  • Batches 5+6 (#13866, #13867): CI all passing, ready for review (30 packages)
  • Strategy fixes (#13872): CI all passing, ready for review (18 packages)
• katello (0/32): Blocked on plugins.

Open PRs

PR	Description	CI
#13866	nodejs batch 5 (16 packages)	all passing
#13867	nodejs batch 6 (14 packages)	all passing
#13872	nodejs strategy fixes (18 packages)	all passing
#13857	rubygem-foreman-tasks	failing
#13862	python-jinja2	failing
#13863	10 python packages	failing
#13813	nodejs tooling	failing
#12872	Build packages with EL10	long-running

Blockers

• rkerberos: New upstream release available that compiles on EL10 (per ekohl). Needs version bump PR.
• newt/radcli: Still need investigation.
• foreman-installer: Waiting on kafo + puppet-agent-puppet-strings.

[Odilhao]

Progress update — ansible-core/ansible-runner chain complete (#24)

Opened #24 as a sub-tracker for the ansible-core/ansible-runner Python dependency chain under foreman_proxy_other_plugin_packages/_tier2 (used by rubygem-smart_proxy_ansible), since it had its own internal ordering (a linear Rust-toolchain sub-chain: setuptools-rust → maturin → cryptography) worth handling as a focused unit rather than folding into the general batch process.

Now closed — all 6 waves merged:

• Wave 1 (#13862, #13863): python-jinja2, python-markupsafe, python-ptyprocess, python-pyparsing, python-six, python-pbr, python-docutils, python-resolvelib, python-semantic-version, python-packaging, python-pexpect
• Wave 2 (#13879): python-daemon, python-lockfile, python-setuptools-scm
• Wave 3 (#13880): python-setuptools-rust
• Wave 4 (#13881): python-maturin
• Wave 5 (#13882): python-cryptography
• Wave 6 (#13883): ansible-core, ansible-runner

foremanctl is also done — it and python-obsah were rebuilt for EL10 in #13853 (merged earlier as part of core-apps).

Bug found and fixed along the way: python-jinja2, python-resolvelib, python-setuptools-scm, ansible-core, and ansible-runner all carried a stray BuildRequires: python%{python3_pkgversion}-rpm-macros (i.e. python3.12-rpm-macros), which doesn't exist on EL10:

No matching package to install: 'python3.12-rpm-macros'

It's redundant — pyproject-rpm-macros (generic) plus python3.12-devel already cover it. Fixed by dropping the line in each spec; worth checking for elsewhere in the manifest if other pyproject-based python packages hit the same wall.

Also hit a transient COPR GPG-signing infra flake on ansible-core's el9 build (gpg: key export failed) — the RPM build itself completed successfully, only the post-build signing step failed. Resolved with a plain CI retrigger, no spec change needed.