Feat/plugin migration cli - #8
Open
ZergsLaw wants to merge 41 commits into
Open
Conversation
…f, err113, forbidigo, mnd, nilerr)
…nd add run-local-test
Split BAZEL_OPTS into startup vs build options so --spawn_strategy=local is not treated as a startup flag (regression from d4d5937). Also set --repo_contents_cache outside /build for Bazel 8/9 compatibility.
Use BAZEL_STARTUP_OPTS/BAZEL_BUILD_OPTS with --spawn_strategy=local and --repo_contents_cache outside the workspace. Copy binaries to /plugin, fix csharp NuGet Version (strip v-prefix), and correct kotlin maven output path.
Skip grpc/go v1.2.0/v1.3.0 (separate-package.patch does not apply) and grpc/web v1.4.2/v1.5.0 (no source tarball in those releases).
Add missing /plugin for six legacy plugins (single-binary COPY or shell wrappers over /app). Hardcode ARG BINARY_NAME defaults so builds no longer depend on plugin.yaml binary metadata.
Cache and normalizeOutput now accept only plugins/.../plugin. Stop auto-passing BINARY_NAME; Dockerfiles own defaults or build_args. Legacy yaml "binary" is accepted and ignored.
Runtime and migrate already use the unified entrypoint name "plugin". Build-time tool names live in Dockerfiles (ARG BINARY_NAME) or build_args.
Artifact is always plugins/{group}/{name}/{version}/plugin; optional
sidecars allowed. Update build/migrate docs accordingly.
When --cfg is set and --plugins-prefix is not explicit, use registry.plugins_dir from the YAML so docker vs local no longer need hand-tuned prefixes. Keep --plugins-prefix as an override. Update Taskfile and add unit tests for resolvePluginsPrefix.
Replace plugins migrate with plugins register. Default scan path is plugins; --cfg still supplies plugins_dir. Add --dry-run (no gRPC), --fail-on-error (default true), and stop the loop immediately on context cancel without continuing. Remove interactive spinner sleep.
Make previously skipped versions buildable instead of excluding them: - grpc/go: inject separate_package via go helper (covers v1.2–v1.6 layouts) - grpc/web v1.4.2/v1.5.0: download prebuilt binaries (no source tarball) - connect-kotlin early releases: GitHub jar / source Dockerfiles - connectrpc/python v0.10.0: package rename to protoc-gen-connectrpc - drop phantom python v0.4.2 (never published as a protoc plugin) - grpc/* Bazel: drop --repo_contents_cache (unsupported on Bazel 7.x)
grpc/* (cpp, csharp, objc, php, python, ruby, web):
- remove --repo_contents_cache from BAZEL_BUILD_OPTS: incompatible with
Bazel 7.1.0/7.6.1 pinned by grpc's own tools/bazel wrapper, was causing
every build in this family to fail instantly with
"Unrecognized option: --repo_contents_cache"
- add --features=-layering_check --host_features=-layering_check: works
around a Bazel/Abseil strict header layering-check false positive
("undeclared inclusion(s)" on absl internals) hit during grpc_plugin_support
- merge the two bazelisk build RUN steps (grpc_plugin_support + language
plugin) into one RUN: each RUN is a separate Docker layer/overlay diff,
and bazel's sandbox trash-dir rename across that layer boundary failed
with "Invalid cross-device link" on the second bazel invocation
(grpc/web only needed the first two fixes, it already builds both
targets in a single bazelisk invocation)
Verified with real forced rebuilds: grpc/cpp:v1.64.0 and grpc/ruby:v1.65.1
now build successfully end-to-end (previously failed instantly).
csharp/objc/php/python received the same fix but are not yet individually
verified end-to-end.
grpc/go:
- drop separate-package.patch and the git apply step entirely; build
vanilla upstream protoc-gen-go-grpc. The patch hardcoded a context line
with the exact version string of the source tag it was generated from,
so it could only ever apply cleanly against that one version and broke
git apply for every other version (most recently confirmed on v1.5.1)
- unskip v1.2.0, v1.3.0, v1.4.0, v1.6.0, v1.6.1 in plugin.yaml: they were
only skipped because the patch didn't apply to their source layout;
without patching there is no such restriction
- verified: all 7 grpc/go versions now build successfully (8-11s each)
Also removed two orphaned, unused patch files that were never referenced
by their Dockerfiles (apple/swift/extramoduleimports.patch,
grpc-ecosystem/gateway/separate_pkg_additional_imports.patch), and the
matching .dockerignore entries.
…o longer patches source)
Each grpc/* Dockerfile was running bazelisk with its default
--jobs=<num CPUs in container>, i.e. every concurrent docker build
tried to grab all 16 host cores for its own clang jobs. Under
`easyp-svc plugins build --parallel 12` this produced load average
~180 on a 16-core host (320+ concurrent clang processes) and 20-min
per-plugin wall times.
Changes (applied identically to all 7 grpc/* Dockerfiles:
cpp, csharp, objc, php, python, ruby, web):
- BAZEL_JOBS=4 ARG (overridable via plugin.yaml build_args), wired
into BAZEL_BUILD_OPTS as --jobs=${BAZEL_JOBS}. Caps per-container
bazel parallelism so N parallel docker builds no longer oversubscribe
the host 10x.
- Bazel --disk_cache and --repository_cache pointed at /build/.cache/*
backed by persistent BuildKit cache mounts:
--mount=type=cache,id=bazel-grpc-disk,target=/build/.cache/bazel-disk
--mount=type=cache,id=bazel-grpc-repo,target=/build/.cache/bazel-repo
--mount=type=cache,id=bazelisk-grpc,target=/build/.cache/bazelisk
Same id across all 7 grpc/* Dockerfiles => grpc_plugin_support
(~1800 abseil+protobuf C++ actions) and the bazelisk-downloaded
bazel binary are shared across languages AND across versions.
Cache is content-addressable, so cross-version sharing is safe
(only matching actions hit; the rest just miss).
- mkdir -p /build/.cache + chown -R nobody:nogroup /build so the
nobody user (uid/gid 65534 in our base image) can write to the
parent dir of the mount points (otherwise bazelisk fails to create
/build/.cache/bazelisk).
- mount uid=65534,gid=65534 matches the nobody user so cache contents
are writable across builds.
Measured on grpc/cpp (real forced rebuilds, no other load on host):
v1.64.0 cold (cache empty) : 13m12s
v1.64.1 warm (after v1.64.0) : 5m36s (~2.4x speedup)
Previously v1.64.0 took ~21m with uncapped jobs and no cache.
protocolbuffers/* uses a different Bazel pin (bazelisk pulls latest,
where --repo_contents_cache IS supported) and is NOT touched here;
it will be migrated separately to preserve --repo_contents_cache.
…cache mounts
Same treatment as for grpc/* in the previous commit, applied to all 9
protocolbuffers/* Dockerfiles that use bazelisk (cpp, csharp, java,
kotlin, objc, php, pyi, python, ruby). (dart/go/js are not bazel-based
and are not touched.)
Changes per Dockerfile:
- BAZEL_JOBS=4 ARG wired into BAZEL_BUILD_OPTS as --jobs=${BAZEL_JOBS}
to cap per-container bazel parallelism (same reasoning as for grpc/*:
N parallel docker builds were oversubscribing the 16-core host).
- Bazel --disk_cache and --repository_cache pointed at /build/.cache/*
backed by persistent BuildKit cache mounts with a dedicated id
namespace (bazel-protoc-*, bazelisk-protoc) so the protoc_lib
target (~1000 C++ actions compiling the protobuf core) is shared
across all 9 languages of the same protobuf version. The first
language builds protoc_lib cold; the other 8 hit the disk cache
for it and only compile their own small plugin target.
- --repo_contents_cache=/tmp/bazel-repo-cache is PRESERVED here
(unlike grpc/* where it had to be removed): protocolbuffers/* uses
bazelisk without a pinned Bazel version, so bazelisk pulls latest
(currently 9.2.0) where the flag IS supported. Removing it would
risk breaking a previously working build.
- mkdir -p /build/.cache + chown -R nobody:nogroup /build so the
nobody user (uid/gid 65534) can write around the mount points.
- mount uid=65534,gid=65534 matches the nobody user.
Measured (real forced rebuilds, no other load on host):
protocolbuffers/cpp:v35.0 cold : 4m40s (22.4MB)
protocolbuffers/csharp:v35.0 warm : 30s (15.0MB, after cpp)
-> ~9.3x speedup on the second language of the same protobuf
version, because protoc_lib is the dominant cost and is
shared across all 9 languages via the cache.
NOTE: very old protobuf releases (e.g. v21.12) fail to build with
current bazelisk-pulled Bazel 9.x (bzlmod can't find @rules_cc via
the legacy WORKSPACE mechanism that those old tarballs expect).
This is a pre-existing issue unrelated to this change - the cache
mount neither fixes nor breaks it. Old versions need a pinned
USE_BAZEL_VERSION or a separate fix, to be handled separately.
Plugins were delivered through a shared ./plugins bind-mount, which does
not work for multi-replica deployments and cannot hold the full registry
(1743 versions, ~7 GB). Artifacts now live in S3-compatible storage and
are materialized on demand.
The unit of delivery is the whole version directory, not a single
binary: the contract allows sidecars next to the entrypoint (connect-kotlin
alone is 355 MB of JVM + jars). Each directory is packed as tar.gz and
stored at {group}/{name}/{version}/plugin.tgz.
Flow: build -> push -> register.
* plugins push packs and uploads archives; only the build pipeline
needs S3 write access.
* register stays metadata-only. The service streams the pushed archive
from storage, computes its sha256 itself and records it in the plugin
config, so a client cannot supply a forged hash. Registering before
pushing fails with FAILED_PRECONDITION, so a registered plugin always
has its artifact.
* At generation time a missing entrypoint triggers a singleflight
download, sha256 verification and atomic unpack before anything is
executed. Storage outages surface as UNAVAILABLE rather than
NOT_FOUND; a checksum mismatch aborts without unpacking.
The gRPC contract is unchanged: no binary payload, no streaming RPC, no
message size limits to raise.
Also fixes pre-existing issues in the storage draft: deprecated endpoint
resolver replaced with BaseEndpoint, typed NotFound detection instead of
string matching, unique temp files (concurrent downloads shared one path
and could interleave), and the ./plugins compose mount switched to rw so
the cache is writable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jj6X26JfaTR8s9mdvp7K2
grpc releases up to v1.55.x pin Bazel 5.3.0 through their //tools/bazel wrapper, and 5.3.0 has no --host_features option (it appeared in Bazel 6.5). The flag was added unconditionally to BAZEL_BUILD_OPTS, so those builds died during flag parsing before compiling anything: bazel version 5.3.0 will be used instead of system-wide bazel ERROR: --host_features=-layering_check :: Unrecognized option This broke all 12 versions in the v1.50.0-v1.55.1 range for every bazel-based grpc plugin (cpp, csharp, objc, php, python, ruby, web) — 72 builds in total. Probe `bazel help build` for the option at build time and pass it only when the pinned toolchain understands it, so one Dockerfile serves both old and new releases. --features=-layering_check is kept unconditional: it is supported all the way back to 5.3.0. Verified against bazelisk: 5.3.0 and 6.0.0 reject --host_features, 6.5.0 and 7.1.0 accept it; the probe reports ABSENT for 5.3.0 and SUPPORTED for 7.1.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013jj6X26JfaTR8s9mdvp7K2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.