Skip to content

chore: modernize and harden codegen toolchain#616

Closed
Devansh-567 wants to merge 11 commits into
p4lang:mainfrom
Devansh-567:chore/modernize-codegen-toolchain
Closed

chore: modernize and harden codegen toolchain#616
Devansh-567 wants to merge 11 commits into
p4lang:mainfrom
Devansh-567:chore/modernize-codegen-toolchain

Conversation

@Devansh-567

@Devansh-567 Devansh-567 commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Pin every moving part in the protobuf code-generation pipeline the base Docker image, Go toolchain, and both protoc-gen-go plugins and align go.mod with the version of Go that is actually being used. The CI script is also tightened so it prints the full diff on failure rather than a generic message.


Problem

The codegen pipeline had four compounding reproducibility issues:

Location Old value Issue
codegen/Dockerfile FROM p4lang/third-party:latest latest is re-tagged on every upstream push; any rebuild could silently pick up a different protoc and produce byte-different output
codegen/Dockerfile ARG GO_VERSION 1.20.5 Go 1.20 reached end-of-life in February 2024; security patches are no longer backported
codegen/Dockerfile protoc-gen-go v1.31 More than a year behind; generates output incompatible with google.golang.org/protobuf v1.34.x
codegen/Dockerfile protoc-gen-go-grpc v1.3 Two minor versions behind v1.5.x; v1.3 was the last release before the --go-grpc_opt=require_unimplemented_servers default changed
go.mod go 1.20 go.mod declared 1.20 while update.sh ran golang:1.20 for go mod tidy; mismatches between the declared minimum and the real toolchain trigger non-deterministic go.sum updates when contributors run go mod tidy locally with a newer Go
codegen/update.sh golang:1.20 Floating minor-version tag golang:1.20 resolves to the newest 1.20.x patch, which was already past EOL
CI/check_codegen.sh [ ! -z "$diff" ] -z is a valid idiom but [ ! -z ] is the double-negative form flagged by ShellCheck SC2236; more importantly the script discards the diff output, making CI failures silent
.github/workflows/codegen.yml ubuntu-latest / actions/checkout@v3 ubuntu-latest switches images without notice; @v3 is deprecated by GitHub

Because latest and floating toolchain tags are resolved at build time, two developers running ./codegen/update.sh a week apart could commit generated files that differ only in formatting or internal constants introduced by a plugin bump causing spurious CI failures for unrelated PRs.


How I found it

Reviewing codegen/Dockerfile directly revealed the three floating references (p4lang/third-party:latest, GO_VERSION=1.20.5, and the untagged @v1.31 / @v1.3 plugin installs). Cross-referencing with the generated file headers in go/p4/v1/p4runtime.pb.go confirmed the mismatch:

// protoc-gen-go v1.31.0
// protoc        v3.18.1

go.mod declaring go 1.20 while update.sh pulling golang:1.20 closed the loop the module minimum and the tidy toolchain were the same stale version, which would silently diverge once a developer with Go 1.22+ ran go mod tidy locally.


Changes

codegen/Dockerfile

  • Base image: replaced p4lang/third-party:latest with a pinned
    sha256 digest. The comment above the FROM line documents how to
    rotate it.
  • Go: bumped GO_VERSION from 1.20.51.25.11 (latest stable
    patch in the current supported 1.25.x series per https://go.dev/dl/).
    Added per-architecture sha256sum verification of the downloaded
    tarball so a supply-chain attack on dl.google.com is caught
    immediately. SHA-256 values are sourced directly from the official
    downloads page and must be updated in lockstep with GO_VERSION.
  • protoc-gen-go: bumped from v1.31v1.34.2.
  • protoc-gen-go-grpc: bumped from v1.3v1.5.1.
  • Changed ARG-based versions for both plugins so they can be overridden
    at docker build time without editing the file.
  • Added rm -rf /var/lib/apt/lists/* after apt-get to reduce image
    size.
  • Changed curl flag from -L to -fsSL (-f fails on HTTP errors,
    -s suppresses progress noise, -S re-enables error messages).

go.mod

  • Bumped go directive from 1.201.25 to match the toolchain
    used in update.sh and codegen/Dockerfile.
  • Replaced the monolithic google.golang.org/genproto dependency (which
    has been deprecated in favour of split sub-modules) with
    google.golang.org/genproto/googleapis/rpc, the only sub-package
    actually imported by the generated files.
  • Bumped google.golang.org/grpc from v1.56.3v1.65.0.
  • Bumped google.golang.org/protobuf from v1.33.0v1.34.2.
  • Removed the now-unused github.com/golang/protobuf indirect
    dependency (superseded by google.golang.org/protobuf).

go.sum

Regenerated to match the updated go.mod. All entries have correct
h1: hashes verified against the Go checksum database.

codegen/update.sh

  • Changed the go mod tidy step from golang:1.20golang:1.25.11
    to keep it in lock-step with the Dockerfile and go.mod.

CI/check_codegen.sh

  • Replaced [ ! -z "$diff" ] with the cleaner [ -n "$diff" ].
  • Switched the diff detection from git status --porcelain to
    git diff --name-only so that the check respects .gitattributes
    line-ending normalization instead of flagging line-ending-only
    differences as content changes.
  • Printed the actual diff output when the check fails, so engineers can
    see exactly what changed without having to re-run locally.
  • Added a success message so it is clear when the check passes in CI
    logs.

.gitattributes (new file)

  • Added to force LF line endings on *.go, *.py, *.sh, *.mod,
    and *.sum files regardless of the contributor's local Git
    core.autocrlf setting. This was required because Windows
    checkouts of this repo previously had core.autocrlf=true
    configured (commonly the default on git for windows installs),
    which silently rewrote LF generated files to CRLF on git add,
    producing noisy diffs unrelated to the actual codegen change.

.github/workflows/codegen.yml

  • Pinned runner from ubuntu-latestubuntu-22.04 for a stable
    environment.
  • Upgraded actions/checkout from @v3@v4.
  • Added --rm to the docker run call that compiles protos (the
    existing call already had it in update.sh but not in the workflow).

How to verify

# Rebuild the image and regenerate outputs locally
./codegen/update.sh
 
# Confirm nothing changed in the working tree
git diff --exit-code go/ go.mod go.sum
git diff --exit-code py/
 
# Or run the full CI check script
./CI/check_codegen.sh

The generated file headers in go/p4/v1/p4runtime.pb.go should now
read:

// protoc-gen-go v1.34.2
// protoc-gen-go-grpc v1.5.1

Checklist

  • Dockerfile base image pinned to digest
  • Go toolchain bumped to supported release (1.25.x) with SHA-256 verification
  • protoc-gen-go and protoc-gen-go-grpc bumped and pinned
  • go.mod minimum version updated to match toolchain
  • Deprecated genproto monorepo dependency replaced with split sub-module
  • go.sum regenerated
  • update.sh golang image pinned to match
  • CI check script hardened and made verbose on failure
  • GitHub Actions runner and checkout action pinned/updated
  • All changes are backward-compatible (no API surface modified)

Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
@Devansh-567

Copy link
Copy Markdown
Contributor Author

Go's 1.22 series topped out at 1.22.12 and has since been removed from the official downloads page entirely, so dl.google.com returned a 404, the tarball was empty/malformed, and the checksum naturally didn't match. I've updated the Dockerfile to use go1.25.11 (current stable) with the correct checksums sourced directly from go.dev/dl, and kept go.mod and update.sh in sync with that same version. Pushing the fix now.

Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
@jafingerhut

Copy link
Copy Markdown
Contributor

I am confused about the whitespace changes in this PR (I am looking at the version of the PR as of commit 9).

When I do the following on a Linux system:

$ git clone https://github.com/p4lang/p4runtime
$ cd p4runtime
$ git log -n 1 | cat
commit ab29f3c2f06bbaeed5d6dec4fe63535eb5cf7840
Author: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Date:   Sat Jun 13 03:41:22 2026 -0400

    Add CI check to validate the presence of copyright and license annotations (#615)
    
    in all files of the repo.  Also add a comment to py/REUSE.toml
    explaining why it exists.
    
    Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>

$ od -c py/REUSE.toml | head
0000000   #       T   h   i   s       f   i   l   e       i   s       u
0000020   s   e   d       b   y       t   h   e       `   r   e   u   s
0000040   e   `       t   o   o   l       f   o   r       c   o   p   y
0000060   r   i   g   h   t       a   n   d       l   i   c   e   n   s
0000100   e  \n   #       v   a   l   i   d   a   t   i   o   n   .  \n
0000120  \n   #       I   n       o   r   d   e   r       f   o   r    
0000140   `   r   e   u   s   e       l   i   n   t   `       t   o    
0000160   p   a   s   s   ,       a   l   l       f   i   l   e   s    
0000200   i   n       t   h   e       r   e   p   o       m   u   s   t
0000220       h   a   v   e  \n   #       c   o   p   y   r   i   g   h

I see that the line endings are the Unix standard newlines, not the Window CR-LF sequence of two characters.

I think that is what we want.

Does your PR try to change the line endings in this file to CR-LF?

If not, then why is the file changed at all?

@jafingerhut

Copy link
Copy Markdown
Contributor

Following up to my previous comment, on the same Linux system of mine I run these commands to check out and examine the same file as it has been modified in the branch of this PR:

$ git clone https://github.com/Devansh-567/p4runtime
$ cd p4runtime
$ git log -n 1 | cat
commit a1911b041e25fe8446fde7cb54242813f046eefd
Author: Devansh-567 <devansh.jay.singh@gmail.com>
Date:   Mon Jun 29 20:44:26 2026 +0530

    fix: use git diff instead of git status in check_codegen for CRLF safety
    
    Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>

$ od -c py/REUSE.toml | head
0000000   #       T   h   i   s       f   i   l   e       i   s       u
0000020   s   e   d       b   y       t   h   e       `   r   e   u   s
0000040   e   `       t   o   o   l       f   o   r       c   o   p   y
0000060   r   i   g   h   t       a   n   d       l   i   c   e   n   s
0000100   e  \r  \n   #       v   a   l   i   d   a   t   i   o   n   .
0000120  \r  \n  \r  \n   #       I   n       o   r   d   e   r       f
0000140   o   r       `   r   e   u   s   e       l   i   n   t   `    
0000160   t   o       p   a   s   s   ,       a   l   l       f   i   l
0000200   e   s       i   n       t   h   e       r   e   p   o       m
0000220   u   s   t       h   a   v   e  \r  \n   #       c   o   p   y

This shows that the PR's modified version of the file has CR-LF line endings.

Why do you want to change text files to have CR-LF line endings?

Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
@Devansh-567

Copy link
Copy Markdown
Contributor Author

Following up to my previous comment, on the same Linux system of mine I run these commands to check out and examine the same file as it has been modified in the branch of this PR:

$ git clone https://github.com/Devansh-567/p4runtime
$ cd p4runtime
$ git log -n 1 | cat
commit a1911b041e25fe8446fde7cb54242813f046eefd
Author: Devansh-567 <devansh.jay.singh@gmail.com>
Date:   Mon Jun 29 20:44:26 2026 +0530

    fix: use git diff instead of git status in check_codegen for CRLF safety
    
    Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>

$ od -c py/REUSE.toml | head
0000000   #       T   h   i   s       f   i   l   e       i   s       u
0000020   s   e   d       b   y       t   h   e       `   r   e   u   s
0000040   e   `       t   o   o   l       f   o   r       c   o   p   y
0000060   r   i   g   h   t       a   n   d       l   i   c   e   n   s
0000100   e  \r  \n   #       v   a   l   i   d   a   t   i   o   n   .
0000120  \r  \n  \r  \n   #       I   n       o   r   d   e   r       f
0000140   o   r       `   r   e   u   s   e       l   i   n   t   `    
0000160   t   o       p   a   s   s   ,       a   l   l       f   i   l
0000200   e   s       i   n       t   h   e       r   e   p   o       m
0000220   u   s   t       h   a   v   e  \r  \n   #       c   o   p   y

This shows that the PR's modified version of the file has CR-LF line endings.

Why do you want to change text files to have CR-LF line endings?

Thanks @jafingerhut for catching this, you were right. My Windows Git config had core.autocrlf silently converting line endings on files I never intended to touch, including py/REUSE.toml. I've reverted py/REUSE.toml, py/LICENSE, py/pyproject.toml, and py/setup.cfg to match main exactly, set core.autocrlf=false globally, and added a .gitattributes to enforce LF on the files this PR actually generates. The diff against main is now scoped only to the intended changes. Sorry for the noise, and thanks again for the careful review.

Signed-off-by: Devansh-567 <devansh.jay.singh@gmail.com>
@Devansh-567
Devansh-567 force-pushed the chore/modernize-codegen-toolchain branch from d871856 to 914a61a Compare June 30, 2026 05:38
@Devansh-567 Devansh-567 closed this Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants