scripts: modernize SBOM generation and workflow

[vivekpatani]

• Generation: syft (pinned v1.44.0 via go install) emits both formats in one pass.
• Augmentation: jq transforms add document metadata and fix UNKNOWN versions for local modules.
• Determinism: byte-stable output (strips timestamps + UUIDs).
• Workflow (mirrors bom_pass): make verify-modern-sbom diffs against committed files; make fix-modern-sbom regenerates.

[k8s-ci-robot]

Hi @vivekpatani. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vivekpatani]

Sample output from a local run

> PASSES="modern_sbom" ./scripts/test.sh
Running with --race
Starting at: Mon Jun 30 13:20:59 PDT 2025

'modern_sbom' started at Mon Jun 30 13:20:59 PDT 2025
validating modern SBOM generation...
syft is already installed: 1.27.1
sbomasm is already installed locally
generating modern SBOM files for version dev
generating SPDX SBOM...
 ✔ Indexed file system                                                                                                                                   . 
 ✔ Cataloged contents                                                                     19be9340167c940e8dc3cbae0b85682c19973b77c6f3a30def20c6976b478d0b 
   ├── ✔ Packages                        [912 packages]  
   ├── ✔ Executables                     [4 executables]  
   ├── ✔ File metadata                   [22 locations]  
   └── ✔ File digests                    [22 files]  
SPDX SBOM generated: /var/folders/6h/5jv5cfs54v72cvvc_pvysdqw0000gn/T/tmp.EUNXl5rXEh/sbom.spdx.json
generating CycloneDX SBOM...
 ✔ Indexed file system                                                                                                                                   . 
 ✔ Cataloged contents                                                                     19be9340167c940e8dc3cbae0b85682c19973b77c6f3a30def20c6976b478d0b 
   ├── ✔ Packages                        [912 packages]  
   ├── ✔ Executables                     [4 executables]  
   ├── ✔ File metadata                   [22 locations]  
   └── ✔ File digests                    [22 files]  
CycloneDX SBOM generated: /var/folders/6h/5jv5cfs54v72cvvc_pvysdqw0000gn/T/tmp.EUNXl5rXEh/sbom.cdx.json
augmenting SBOM files with enhanced metadata...
augmenting SPDX SBOM...
{"level":"info","ts":1751314862.2400372,"caller":"edit/spdx_edit.go:82","msg":"SPDX error updating supplier: not supported"}
{"level":"info","ts":1751314862.2400792,"caller":"edit/spdx_edit.go:82","msg":"SPDX error updating repository: not supported"}
SPDX SBOM augmented successfully
augmenting CycloneDX SBOM...
CycloneDX SBOM augmented successfully
SBOM augmentation completed successfully
modern SBOM files generated successfully
files created:
  - /var/folders/6h/5jv5cfs54v72cvvc_pvysdqw0000gn/T/tmp.EUNXl5rXEh/sbom.spdx.json
  - /var/folders/6h/5jv5cfs54v72cvvc_pvysdqw0000gn/T/tmp.EUNXl5rXEh/sbom.cdx.json
validating generated SBOM files...
Validated: /var/folders/6h/5jv5cfs54v72cvvc_pvysdqw0000gn/T/tmp.EUNXl5rXEh/sbom.spdx.json
Validated: /var/folders/6h/5jv5cfs54v72cvvc_pvysdqw0000gn/T/tmp.EUNXl5rXEh/sbom.cdx.json
all SBOM files validated successfully
modern SBOM generation validation PASSED
'modern_sbom' PASSED and completed at Mon Jun 30 13:21:02 PDT 2025
SUCCESS

[jmhbnz]

Awesome progress @vivekpatani - thanks for drafting this!

[jmhbnz]

/ok-to-test

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.70%. Comparing base (dcadd27) to head (9cb2596).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

see 19 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #20303      +/-   ##
==========================================
- Coverage   69.73%   69.70%   -0.03%     
==========================================
  Files         449      449              
  Lines       38177    38177              
==========================================
- Hits        26622    26613       -9     
- Misses      10127    10137      +10     
+ Partials     1428     1427       -1     

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dcadd27...9cb2596. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vivekpatani]

Hello team, gentle bump @jmhbnz @ivanvc @ahrtr @serathius

[ivanvc]

Thanks for putting this together, @vivekpatani, great job :)

I ran scripts/generate-modern-sbom.sh v3.6.1 locally, but the output file shows UNKNOWN for our local packages both in sbom.cdx.json and sbom.spdx.json. And in sbom.spdx.json, all the dependencies show NOASSERTION for licenseDeclared and copyrightText (among other fields). I assume this is not the expected behavior, right? Unless I executed them incorrectly.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[vivekpatani]

@ivanvc not inline but the response to the above comment:

• UNKNOWN versions: These were the local workspace modules (go.etcd.io/etcd/api/v3, client/v3, etc.) that use replace directives pointing to local paths. Syft can't resolve their versions from source. The script now post-processes with jq to replace all UNKNOWN versions, I think seems like a decent approach?
• NOASSERTION for licenseDeclared: This is correct SPDX behavior. licenseDeclared means "what the package author explicitly declared", syft can't determine that from source scanning. The actual license detection is in licenseConcluded, where syft reports licenses for ~609/923 packages.

Sorry it took so long @ahrtr @ivanvc feedback, would be appreciated.

[vivekpatani]

Hello,

IIUC, putting syft in tools/mod is what's breaking the verify job. syft drags in way too much, and one of its deps collides with the version the rest of etcd uses. fix is to give syft its own module that's outside the workspace.

pull-etcd-verify runs make verify, and it dies on the verify-dep step. that step has one rule: every dependency has to be the same version across all modules in the workspace. syft v1.45.1 pulls in containerd, which wants a newer protobuf than we pin:

• tools/mod (via syft): google.golang.org/protobuf v1.36.12-0.2026...
• everything else: google.golang.org/protobuf v1.36.11

two protobuf versions in one workspace → verify-dep says no → job fails.

syft isn't a normal tool. our other tools (golangci-lint, cfssl, etc.) happen to use deps that line up with ours. syft doesn't, it has ~463 deps and needs newer versions than we pin. and because tools/mod is part of the go workspace, it's forced to agree with everything else. it can't.

I think the fix would be(?):

• new tools/sbom/go.mod with just syft
• keep it out of go.work (and out of update_go_workspace.sh, otherwise make fix quietly re-adds it and we're back here)
• install it via go install from that module
• add /tools/sbom to dependabot

As an aside, the govulncheck failure isn't from this PR. govulncheck is installed @latest (scripts/test.sh), and it rolled from v1.3.0 (x-tools v0.44.0) to v1.4.0 (x-tools v0.46.0), which panics on a generic in github.com/olekukonko/ll (pulled in via tablewriter, used by etcdctl/etcdutl/tests). #21971 passed govulncheck on Jun 16 with v1.3.0 and the same olekukonko/ll v0.1.6; this PR hit v1.4.0 on Jun 17 and panicked. it'll hit main and other PRs once they're retested, so we should pin govulncheck instead of @latest?

@ivanvc @serathius @ahrtr

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ivanvc, vivekpatani

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ivanvc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ahrtr]

Pls resolve the workflow pull-etcd-verify failure, thx

[vivekpatani]

@ivanvc @ahrtr @serathius I tried to resolve etcd-pull-verify however it's a bit more nuanced.

I tried this exactly as suggested first: syft was added to tools/mod and installed via run_go_tool, like every other tool (that's what was in the commit that ran CI). It's what surfaced the pull-etcd-verify failure, and digging in, syft turns out to be a special case.

tools/mod is part of go.work, and verify-dep (dep_pass in scripts/test.sh) enforces a single version per dependency across all workspace modules. syft v1.45.1 pulls ~280 transitive modules, and several of them require versions newer than the rest of etcd pins:

• google.golang.org/protobuf v1.36.12-0.2026... vs our v1.36.11
• go.opentelemetry.io/otel (+ /metric, /sdk, /sdk/metric, /trace)
• github.com/prometheus/procfs, golang.org/x/exp

Two versions of the same dep in one workspace makes verify-dep fail, by design. This is unique to syft: golangci-lint, cfssl, gomodguard, etc. happen to align with our pinned versions, but syft is a full application (containerd, AWS SDK, the charm TUI stack, …) on a fast-moving dependency tree, so it can't be held to the workspace's versions.

The fix keeps the intent of your guidance (install via go install, version tracked by Dependabot) but moves syft to its own module, tools/sbom, kept out of go.work. Because verify-dep only inspects workspace modules, syft's deps no longer collide, and the rest of the workspace stays byte-identical to main (zero delta). I also excluded tools/sbom from update_go_workspace.sh so make fix won't silently re-add it, and added a /tools/sbom Dependabot entry.

So: agreed on go install + Dependabot, the only deviation is workspace membership, which is forced by syft's dependency footprint. Open to alternatives if you'd prefer (e.g. pinning syft to a version whose deps align), but that seems more fragile across future bumps.

[vivekpatani]

/retest

[ahrtr]

I am curious why not to add this into tools/mod/tools.go? why add a new mod and file?

[ahrtr]

same question as #20303 (comment)

[ahrtr]

This isn't specific to syft, all tools defined in tools/mod have the same "problem". So we may still want to add syft in tools/mod

[ahrtr]

tools/mod is part of go.work, and verify-dep (dep_pass in scripts/test.sh) enforces a single version per dependency across all workspace modules. syft v1.45.1 pulls ~280 transitive modules, and several of them require versions newer than the rest of etcd pins:

We should manage all tools in a consistent way. I think we have two options:

• move tools/mod out of go.work, and add syft into tools/mod
• add syft into tools/mod, and ensure we have consistent version for all deps

either way, we need to add syft into tools/mod. cc @ivanvc @vivekpatani

[ahrtr]

I was curious why the two BOM files (sbom.spdx.json and sbom.cdx.json) are so huge files. After looking into the contents of the two files, I understand that for each dependency syft generate an item for each etcd module.

For example, github.com/spf13/cobra is used by 6 etcd modules (not including the modules under tools), so there are 6 items for such single dependency in both files. Since etcd already guarantees that all etcd modules have consistent dependency versions, so I think it doesn't make sense to have duplicated items for each dependency.

Also for github action components such as actions/checkout, there are also multiple entries (such as 6 entries for this example) in both files, because it's used in 6 github workflow yaml files. Again, it doesn't make sense to have duplicated items.

@vivekpatani can you please resolve this to remove the duplicated items for each dependency?

[vivekpatani]

We should manage all tools in a consistent way. I think we have two options:
move tools/mod out of go.work, and add syft into tools/mod
add syft into tools/mod, and ensure we have consistent version for all deps
either way, we need to add syft into tools/mod

Agreed on managing tools consistently. Quick take on the two options:

• Option 2 (add to tools/mod + force consistent versions): I'd push back on this one. I tested it. Making verify-dep pass requires go work sync, which rewrites server/go.mod and the root module onto syft's pre-release google.golang.org/protobuf v1.36.12-0.2026 (we're on the released v1.36.11). That ties the server's protobuf version to an SBOM tool, and every syft bump could churn production deps. To reproduce:

( cd tools/mod && go get github.com/anchore/syft@v1.45.1 ) && go work edit -go 1.26.3 && go work sync
grep protobuf server/go.mod   # now v1.36.12-pre; 

• Option 1 (move tools/mod out of go.work + add syft): viable, but a wider blast radius. It does fix the conflict (verify-dep stops governing tools/mod, and the go 1.26.3 vs go.work go 1.26 issue goes away), and it would let us install syft via run_go_tool like the other tools, which addresses @ivanvc earlier point. But it changes dependency governance for every tool in tools/mod, likely needs run_go_tool/tool_get_bin to run with GOWORK=off, and could shift the other tools' shared deps via MVS inside tools/mod. I'd want to validate those ripples first.

For context, tools/sbom as it stands is basically Option 1 scoped to just syft: out of the workspace, GOWORK=off, no impact on the other tools, and it still gets Dependabot tracking through its own stanza.

I'm happy either way. If you'd prefer a single tools module, I'll take Option 1 and validate the tool-resolution and MVS ripples. If the scoped module is acceptable, I'll keep it. What's your preference? @ahrtr @ivanvc

[vivekpatani]

Q (@ahrtr): "syft generates an item for each dependency per etcd module (cobra has 6 items), and multiple entries for github actions like actions/checkout (6 entries). Since etcd guarantees consistent versions, duplicates don't make sense. Can you remove the duplicated items?"

Good catch, confirmed it. The dir scan catalogs each module separately, so a dep shared by N modules shows up N times. SPDX is 628 packages but only 115 unique name@version (82% redundant), CDX is the same (627 to 114). cobra is 6x, actions/checkout is 6x, and the test-only deps (testify, go-spew, yaml.v3) are 10x since they sit in all 10 modules.

Two ways to fix it:

1. jq dedup on the current source scan: collapse to one entry per name@version and rewrite the SPDX relationships / CDX dependencies to the canonical refs. Keeps the existing deterministic pipeline, just need to avoid orphaning any relationship refs.
2. Scan the built binaries instead of dir:.: this dedups naturally and drops the test-only deps that shouldn't be in a release SBOM anyway. Cleaner, but a bigger change, and I'd re-confirm determinism holds for binary scans.

I'm leaning toward (1) for this PR since it's the smaller change and preserves determinism, with (2) as a possible follow-up. Sound good? @ahrtr

[ahrtr]

Option 1 (move tools/mod out of go.work + add syft): viable, but a wider blast radius.

Can you explain why there is a wider blast radius?

I'm happy either way. If you'd prefer a single tools module, I'll take Option 1 and validate the tool-resolution and MVS ripples. If the scoped module is acceptable, I'll keep it. What's your preference? @ahrtr @ivanvc

I'd defer to you and @ivanvc to work out the best solution. We need to setup the requirement first,

• we should have consistent way to mange all tool
• We should ensure all etcd production modules have consistent versions of deps.
• Do we need to ensure the deps used by tools have consistent version as production modules? It seems that we can simplify thing if we don't require this?
• should we get all dependencies used by tool included in the BOM files? How does K8s handle it?

Two ways to fix it:

1. jq dedup on the current source scan: collapse to one entry per name@version and rewrite the SPDX relationships / CDX dependencies to the canonical refs. Keeps the existing deterministic pipeline, just need to avoid orphaning any relationship refs.
2. Scan the built binaries instead of dir:.: this dedups naturally and drops the test-only deps that shouldn't be in a release SBOM anyway. Cleaner, but a bigger change, and I'd re-confirm determinism holds for binary scans.

I'm leaning toward (1) for this PR since it's the smaller change and preserves determinism, with (2) as a possible follow-up. Sound good? @ahrtr

These seem workaround solutions. Does syft have native way to handle this?

[vivekpatani]

Can you explain why there is a wider blast radius?

It is the proto toolchain. protoc-gen-go ships in the same module as google.golang.org/protobuf, and it is the binary genproto.sh builds via tool_get_bin (scripts/genproto.sh:80). Adding syft to tools/mod forces that module's protobuf to syft's pre-release v1.36.12, so protoc-gen-go's stamp flips from v1.36.11 to v1.36.11-devel, and regen would restamp all 12 committed *.pb.go (verify-genproto fails). Repro:

WT=$(mktemp -d); git worktree add -q --detach "$WT" HEAD
GOWORK=off go -C "$WT/tools/mod" get github.com/anchore/syft@v1.45.1
GOBIN="$WT/bin" GOWORK=off go -C "$WT/tools/mod" install google.golang.org/protobuf/cmd/protoc-gen-go
"$WT/bin/protoc-gen-go" --version   # v1.36.11-devel  (repo's *.pb.go are stamped v1.36.11)
git worktree remove --force "$WT"

Do we need to ensure the deps used by tools have consistent version as production modules? It seems that we can simplify things if we don't require this?

No, we do not need to. Tools are build-time only and never ship, so their deps do not have to match the production modules. That is exactly what makes isolating syft acceptable: its deps conflict with the workspace, but since tool deps are not required to be consistent with production, keeping it out of go.work is correct, not a workaround. Tool management stays consistent (go install + Dependabot, the etcd convention); only the module location differs.

should we get all dependencies used by tool included in the BOM files? How does K8s handle it?

No. The BOM should describe what ships, not the build tools, so tool deps are excluded (the script already excludes ./tools/**). k8s does the same: it uses its own tool bom (sigs.k8s.io/bom, not syft) to scan the source repo for the release SBOM, and does not put build-tool deps in the product SBOM.

These seem workaround solutions. Does syft have native way to handle this?

Not via a flag. syft has no dedup/merge option, and syft edit is still an open feature request (anchore/syft#3485,anchore/syft#4461); its only built-in dedup is binary-vs-OS overlap, which does not apply to a Go source tree. The native way to avoid the per-module duplication is to scan the built binaries instead of dir:. (one module graph per binary, deduplicated by construction):

scan	total	unique	duplicates
dir:. (current)	628	115	513
binaries (etcd 60, etcdctl 38, etcdutl 57)	153	78	0 per-binary

The binary scan still captures every shipped dep (grpc, protobuf, bbolt, raft, zap, otel) and drops the test-only and CI deps that do not ship.

[ahrtr]

That is exactly what makes isolating syft acceptable

OK.

The native way to avoid the per-module duplication is to scan the built binaries

Wouldn't that have the platform-dependent problem introduced at build time due to build tag such as //go:build cgo && amd64, as mentioned in #18902 (comment)? Also if we scan binaries, does it mean we ignore all dependencies used by e2e test?

(anchore/syft#3485,anchore/syft#4461)

I see jq is recommended in anchore/syft#4324 (comment), so I am not against your previous proposal of using jq.

[vivekpatani]

Wouldn't that have the platform-dependent problem introduced at build time due to build tag such as //go:build cgo && amd64, as mentioned in #18902 (comment)? Also if we scan binaries, does it mean we ignore all dependencies used by e2e test?

about the platform bit, pinning the build (CGO_ENABLED=0 GOOS=linux GOARCH=amd64 -trimpath) makes it reproducible on any host, so the build-tag concern is avoidable.

But your second point is a valid concern, a binary scan drops the test/e2e deps (testify, antithesis-sdk-go, gofail,...) that bill-of-materials.json tracks, so the scan stays dir:.. I've implemented the jq dedup on the dir:. output: SPDX 628 -> 114 unique, CycloneDX 627 -> 114, GitHub Actions collapsed too, 0 duplicates.

Comparison and raw list are in the gist.

[ahrtr]

/retest

[vivekpatani]

/retest

[ivanvc]

Good catch, confirmed it. The dir scan catalogs each module separately, so a dep shared by N modules shows up N times. SPDX is 628 packages but only 115 unique name@version (82% redundant), CDX is the same (627 to 114). cobra is 6x, actions/checkout is 6x, and the test-only deps (testify, go-spew, yaml.v3) are 10x since they sit in all 10 modules.

Is this dedup process something that other projects do?

---

Regarding the issue with the tools directory. I suggested removing tools/mod from the verify-dep check before.

I agree that we should have a better way of managing the tools. But maybe rather than introducing a new module in this PR. Could we just accept installing the tool with a fixed version from the script? Adding a new tool pattern may make it more difficult to remove it later.