[CI] Collapse the ~419-job pipeline into a shared-workspace design#2297
Open
chr-hertel wants to merge 5 commits into
Open
[CI] Collapse the ~419-job pipeline into a shared-workspace design#2297chr-hertel wants to merge 5 commits into
chr-hertel wants to merge 5 commits into
Conversation
Every package was installed on its own, so a PR spent ~419 jobs and ~600 composer installs. PHPStan and PHPUnit only need an autoloader, and composer resolves the same versions for the intersection of all package constraints as it does per package -- so both now run against one merged install per resolution axis. .github/build-workspace.php generates a root composer.json requiring every package via path repositories, plus the union of their third-party require/require-dev and their autoload-dev namespaces. The per-package vendor/ is reduced to the two paths the configs actually resolve (vendor/autoload.php, vendor/phpstan), which avoids the symlink cycle that made isolate-bridge necessary. Granularity is kept: .github/run-in-packages.sh runs every package, groups its output, and reports each failing package instead of stopping at the first one. PHPStan additionally annotates failures inline at file:line. --prefer-lowest cannot share a workspace: a merged install resolves to the highest of all per-package floors (Mate allows symfony/console ^5.4, Platform requires ^7.3), so the lowest matrix keeps installing each package in isolation. Also drops .github/workflows/.utils.sh, which nothing referenced. 419 -> 133 jobs.
Member
Author
|
this approach is 42% slower (according to a single run), but maybe parallelization within the jobs/steps might help |
Collapsing ~419 jobs into 133 traded parallelism for job count: code-quality's wall clock went from 185s to 263s, since each shard analysed its packages serially. Run them in parallel instead, buffering each package's output and printing it in input order so the log groups and per-package failure attribution are unchanged. This is what symfony/phpunit-bridge's simple-phpunit does for symfony/symfony. On four cores, the Platform Bridges PHPStan shard drops from 89s to 36s, and the full 93-package PHPUnit sweep from 36s to 17s.
Member
Author
|
parallelization basically cuts it to 50% 💪 |
The Sqlite bridge tests only load the extension on PHP >= 8.4 and fall back to a
plain PDO connection otherwise, so installing it on the PHP 8.2 jobs never had an
effect. The integration job installed it too, although Sqlite declares no
#[Group('integration')] test and therefore never enters that matrix.
lyrixx
approved these changes
Jul 10, 2026
lyrixx
left a comment
lyrixx
left a comment
Member
There was a problem hiding this comment.
It looks much simpler, and cleaner. Thanks !
PHPStan compiles its DI container into $TMPDIR/phpstan, keyed by a hash of the
configuration. 32 of the 37 platform bridges ship a byte-identical
phpstan.dist.neon, so running them in parallel made them race on the very same
container file and the Platform Bridges shard failed with:
ContainerLoader.php line 80:
Unable to include '/tmp/phpstan/cache/nette.configurator/Container_2b467e6b85.php'
Point each process at its own temporary directory. The container cache is no
longer shared, which costs ~5s on the largest shard and removes the race.
chr-hertel
force-pushed
the
optimize-pipeline
branch
from
f2e9042 to
d2ff1fc
Compare
chr-hertel
changed the title
The lowest matrix was the last per-package sprawl: 8 package jobs + 85 bridge jobs,
each paying ~24s of fixed setup (checkout, PHP, root install, build-packages) to
run a ~1s test suite.
A merged install can't be shared here -- composer would resolve the intersection
of all constraints and pick the highest of every package's floor, never exercising
the lowest bound a single package declares. So each package still installs on its
own, but grouped by component into 6 jobs driven by .github/run-lowest.sh.
Installs run sequentially against the shared, warm download cache: parallel composer
processes race while creating cache entries, and that cache is what keeps the lowest
resolution fast. The biggest group (37 platform bridges) takes ~76s, well under the
pipeline's critical path.
Components install in place. Bridges can't -- a bridge sits inside its component's
path-repository tree and installing it there makes composer chase a symlink cycle --
so each bridge is copied to a scratch tree that keeps it at its original depth under
the repository root (".lowest/" replaces the leading "src/"), so relative paths to
the shared fixtures/, .phpstan/ and sibling test suites still resolve. This is the
same reason .github/actions/isolate-bridge relocates bridges to tmp/<component>/...
build-matrix.yaml now only computes the store integration list; the packages and
bridges matrices it used to emit are no longer consumed.
93 lowest jobs -> 6.
chr-hertel
force-pushed
the
optimize-pipeline
branch
from
d2ff1fc to
e348184
Compare
Member
Author
|
@OskarStark we're losing here granularity on first level, but still have it on step level in case a job fails: 
hard feelings about this rework? not sure tho if all combinations of change scenarios will work. claude ran some tests for change scenarios on my fork:
(and nvm the fabbot failures) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI installed every package on its own — ~419 jobs and ~600
composer installs per PR. PHPStan and PHPUnit only need an autoloader, so most of that install work was redundant. This reworks the pipeline around a single shared install per resolution axis.Changes
.github/build-workspace.phpgenerates one rootcomposer.json(all 93 packages via path repos + the union of their third-partyrequire/require-dev+ theirautoload-dev). Both tools run against that single install.vendor/is reduced to the two paths the configs resolve (autoload.php,phpstan), avoiding the symlink cycleisolate-bridgeused to work around..github/run-in-packages.shruns every package in parallel with buffered, grouped output and per-package failure attribution; PHPStan failures annotate inline at file:line.--prefer-lowestjobs — the lowest matrix (which genuinely can't share an install) collapses from 93 jobs into 6 component groups via.github/run-lowest.sh; components install in place, bridges are copied to a depth-preserving scratch tree, installs run serially against the warm cache.TMPDIR(PHPStan's container cache is shared and 32 bridges have byte-identical configs).sqlite-vecinstalls that couldn't take effect (bridge loads it only on PHP ≥ 8.4; integration job's Sqlite never enters that matrix), the unreferenced.github/workflows/.utils.sh, and the unusedbuild-matrix.yamloutputs (now only the store-integration list).Before / after
composer installsKept per-package (on purpose)
--prefer-lowest: a merged install resolves to the highest of all per-package floors (Mate allowssymfony/console ^5.4, Platform requires^7.3), so it's grouped but never merged.