Skip to content

refactor(protoc): migrate to protocompile experimental compiler with embedded proto deps#404

Open
Kybxd wants to merge 5 commits into
masterfrom
edition-2024
Open

refactor(protoc): migrate to protocompile experimental compiler with embedded proto deps#404
Kybxd wants to merge 5 commits into
masterfrom
edition-2024

Conversation

@Kybxd
Copy link
Copy Markdown
Collaborator

@Kybxd Kybxd commented May 13, 2026

Why

Protobuf Edition 2024 is not understood by bufbuild/protocompile's
stable Compiler. To unblock Edition 2024 schemas in this codebase, we
follow the same path Buf itself took in
buf v1.69.0 and
switch the internal protoc wrapper to protocompile's experimental
compiler, which has Edition 2024 support.

What changes vs. the stable compiler

The experimental compiler exclusively consumes source files via
source.Opener. It no longer accepts pre-compiled FileDescriptors
through protoregistry.GlobalFiles — which is how the stable compiler
used to pick up tableau/protobuf/*.proto and
buf/validate/validate.proto. We need a new mechanism to make those
imports resolvable at compile time.

How

Vendor the required .proto files under internal/x/xproto/protoc/embedded/
and expose them via go:embed:

Embedded file Source
tableau/protobuf/tableau.proto copied from proto/tableau/protobuf/
tableau/protobuf/wellknown.proto copied from proto/tableau/protobuf/
buf/validate/validate.proto buf export of buf.build/bufbuild/protovalidate at the commit pinned in buf.lock

Standard imports (google/protobuf/*) continue to be served by
source.WKTs and need no embedding.

Refresh workflow

A go:generate directive in embed.go invokes gen_embedded.sh, which:

  1. Wipes embedded/ and recreates embedded/tableau/protobuf/.
  2. Copies proto/tableau/protobuf/*.proto (top-level only — matches
    buf.yaml excludes for internal/ and unittest/).
  3. Reads the protovalidate commit out of buf.lock and forwards it to
    buf export buf.build/bufbuild/protovalidate:<commit>.

⚠️ buf export <remote-module> does not consult the workspace's
buf.lock — that file only governs intra-workspace transitive
resolution. Without an explicit commit suffix, buf export would pull
the latest published commit. The script reads the commit from
buf.lock so the vendored copy stays in sync automatically.

Why is embedded/ checked into version control?

So that downstream consumers of
go install github.com/tableauio/tableau/cmd/tableauc@latest can build
without buf installed and without running go generate
proxy.golang.org only packages git-tracked files.

Upgrading protovalidate

buf dep update          # bumps buf.lock
go generate ./...       # refreshes embedded/
git add -A && git commit

Kybxd added 3 commits May 13, 2026 18:01
The experimental compiler only accepts source files via source.Opener, so
vendor required .proto deps under internal/x/xproto/protoc/embedded/ and
expose them through go:embed:

  - tableau/protobuf/*.proto: copied from proto/tableau/protobuf/
  - buf/validate/validate.proto: exported from buf.build/bufbuild/protovalidate
    at the commit pinned in buf.lock

gen_embedded.sh (invoked via go:generate) refreshes the embedded tree and
explicitly forwards buf.lock's protovalidate commit to `buf export`, since
ad-hoc `buf export` does not consult buf.lock. The embedded/ tree is
committed so `go install ...@latest` works without buf or go generate.
@codecov
Copy link
Copy Markdown

codecov Bot commented May 14, 2026

Codecov Report

❌ Patch coverage is 60.24096% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.44%. Comparing base (99d72fc) to head (227b52c).

Files with missing lines Patch % Lines
internal/x/xproto/protoc/compiler.go 58.46% 18 Missing and 9 partials ⚠️
internal/protogen/protogen.go 66.66% 2 Missing and 2 partials ⚠️
internal/x/xproto/protoc/embed.go 60.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #404      +/-   ##
==========================================
+ Coverage   73.38%   74.44%   +1.05%     
==========================================
  Files          88       89       +1     
  Lines       11300     9154    -2146     
==========================================
- Hits         8293     6815    -1478     
+ Misses       2440     1762     -678     
- Partials      567      577      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant