Buf proto update — v1.30.1 read-path UUID fix (Phases 19–22)

.planning/PROJECT.md

@@ -84,5 +84,9 @@ The proxy must correctly serve both old (v1.30.1) and modern (v1.69.0+) Buf CLI
 
 This document evolves at phase transitions and milestone boundaries.
 
+## Current State
+
+Phase 22 complete (2026-07-08) — v1.30.1 v1alpha1 read-path now resolves 32-char buf UUIDs to 40-char git SHAs via the Phase 18 probe ladder before provider GetFiles calls. PR-22-1/2/3/5 unit-verified. PR-22-4 live e2e (TestGenerateWithPinnedBufLock/v1.30.1) blocked by provider-transport TLS issue to raw.githubusercontent.com (handler fix independently proven) — tracked in 22-HUMAN-UAT.md. Code review (22-REVIEW.md) flagged CR-01 stale-digest + 6 warnings as future-hardening candidates.
+
 ---
-*Last updated: 2026-07-06 after v1.3 milestone completion (Phases 11–16, including Phase 16 commit-id format cutover)*
+*Last updated: 2026-07-08 after Phase 22 completion (v1.30.1 v1alpha1 read-path UUID resolution)*

.planning/ROADMAP.md

@@ -41,6 +41,9 @@
 - [ ] **Phase 15: Operational Logging** — Panic recovery middleware with full stack trace
 - [x] **Phase 16: Commit ID Resolution Improvements** — Use first 16 bytes of git SHA as commit id (incl. short-sha support), probe all configured repos on cache miss, clearer not-found error response and log (completed 2026-07-06)
 - [x] **Phase 17: Fix PR #37 review findings** — Address pre-merge review findings from PR #37 (Phase 16) so the commit-id format cutover lands without undoing the v1.3 logging-quality work or breaking Bitbucket SHA-256 repositories (completed 2026-07-07)
+- [x] **Phase 18: Respect buf.yaml dependency refs** — Honor `Name.ref` end-to-end + isSHA-gated provider ref-resolution + prewarm removal with `commitUUIDInverse`-based probe (completed 2026-07-07)
+- [x] **Phase 19: e2e tests for ref-honoring in buf.yaml deps** — Three `TestRefRespected_*` tests that exercise the proxy's ref-honoring behavior end-to-end via a real buf CLI + real GitHub API; tests pass with `EASYP_GH_TOKEN` set, skip cleanly otherwise (completed 2026-07-07)
+- [ ] **Phase 20: Fix ref-honoring regressions discovered in Phase 19** — Restore the no-ref (HEAD) path that Phase 18 broke for buf v1.30.1, and fix the v1beta1 path where the ref is being ignored on `buf dep update`
 
 ## Phase Details
 
@@ -190,6 +193,51 @@ Plans:
 
 - [x] [18-01](./phases/18-respect-buf-yaml-dependency-refs-not-always-head-fix-related/18-01-PLAN.md) — Honor `Name.ref` end-to-end + isSHA-gated provider ref-resolution + prewarm removal with `commitUUIDInverse`-based probe
 
+### Phase 19: we need e2e tests for the ref specified for dependency - looks like it does not work
+
+**Goal:** End-to-end proof that the proxy honors the `ref` field in a `buf.yaml` dependency: a `buf mod update` (and `buf dep update` on v1.32+) run with a `name:ref` dep in buf.yaml must produce a `buf.lock` whose `commit:` line is the UUID derived from the SHA at that ref, not from HEAD. Phase 18 (commit 4fc6f28) shipped the ref-honoring code path (`parseResourceRefName` reads proto field 3, `ServeHTTP`/`ServeGraph` pass `ref.ref` to providers, providers gate on `isSHA(commit)` and route refs through their commit-fetch APIs). This phase adds the missing real-server e2e tests that exercise the integration with a real buf CLI + real GitHub API.
+**Depends on:** Phase 18
+**Requirements**: SC-19-1, SC-19-2, SC-19-3 (all derived from this plan; see 19-01-PLAN.md)
+**Success Criteria** (what must be TRUE):
+
+  1. For every cached buf binary in `testdata/buf/`, `buf mod update` against the proxy with a `name:ref` dep in buf.yaml pins `buf.lock` to a different commit than the no-ref run; a regression where the proxy returns HEAD for the ref-pinned run fails the test with both lock files shown side-by-side (`TestRefRespected_ModUpdate_DiffersFromHead`)
+  2. For v1.69.0, `buf mod update` with the pinned googleapis tag (`common-protos-1_3_1`) pins `buf.lock` to the UUID derived from the SHA at that tag — the SHA is fetched via `git ls-remote https://github.com/googleapis/googleapis refs/tags/common-protos-1_3_1` and the UUID is computed via the same `commitUUID` byte table the proxy uses (`TestRefRespected_ModUpdate_MatchesUpstreamSHA`)
+  3. For v1.69.0, `buf dep update` with the pinned ref pins `buf.lock` to a different commit than the no-ref run; the modern subcommand exercises the same ref-honoring path the deprecated `buf mod update` does, on the v1beta1 protocol (`TestRefRespected_DepUpdate_DiffersFromHead`)
+
+**Plans:** 1 planPlans:
+
+- [ ] [19-01](./phases/19-we-need-e2e-tests-for-the-ref-specified-for-dependency-looks/19-01-PLAN.md) — Adopt in-progress e2e drafts (ref_test.go + testutil/server.go additions); 2 tasks: (1) finalize testutil additions and verify testutil unit tests still pass; (2) finalize e2e tests, verify they compile + list + skip cleanly without EASYP_GH_TOKEN, and commit both files
+
+### Phase 20: fix the problem with refs discovered on phase 19
+
+**Goal:** [To be planned]
+**Requirements**: TBD
+**Depends on:** Phase 19
+**Plans:** 0 plans
+Plans:
+
+- [ ] TBD (run /gsd-plan-phase 20 to break down)
+
+### Phase 21: we need another e2e test: we are doing buf generate with buf.lock pointing to the valid bt not the latest commit. This is valid situation and should work with v1 and v2
+
+**Goal:** Add a single e2e test (`TestGenerateWithPinnedBufLock`) that proves the proxy can serve a `buf generate` request when `buf.lock` pins a valid (but non-HEAD) commit. The test runs for every cached buf version (v1.30.1 v1alpha1 + v1.69.0 v1beta1), derives the pinned UUID from a real googleapis tag via `git ls-remote` + the test-side `commitUUIDForTest` byte-table mirror, overwrites `buf.lock`'s `commit:` line with the pinned UUID, and asserts `buf generate` exits 0 + generates at least one non-empty `gen/go/google/type/*.pb.go` file whose content contains `package google.type`. This is the regression guard for the Phase 18/20 read-path (infoCache writeback + post-restart `probeCommitID`): a future regression that 400s the proxy on a pre-existing UUID or serves wrong content is caught at CI time.
+**Requirements**: SC-21-1, SC-21-2 (derived from this plan; see 21-01-PLAN.md)
+**Depends on:** Phase 20
+**Plans:** 1/1 plans complete
+Plans:
+
+- [x] [21-01](./phases/21-we-need-another-e2e-test-we-are-doing-buf-generate-with-buf-/21-01-PLAN.md) — Adopt new e2e drafts (generate_test.go + testutil/server.go additions); 2 tasks: (1) add `RunBufGenerateWithPinnedLock` public wrapper + `runBufGenerate` private helper in testutil/server.go (writes buf.yaml + buf.gen.yaml + dummy.proto, runs `buf mod update`, overwrites the lock's `commit:` line via `strings.Replace(..., 1)`, then runs `buf generate`); (2) add `e2e/generate_test.go` with `TestGenerateWithPinnedBufLock` matrix test, verify it compiles + lists + skips cleanly without `EASYP_GH_TOKEN`, commit both files
+
+### Phase 22: Fix v1.30.1 v1alpha1 read-path UUID handling; verify v1 protocol works
+
+**Goal:** Wire the v1alpha1 `DownloadManifestAndBlobs` handler on `*api` to the existing Phase 18 UUID-resolution machinery on `*commitServiceHandler` (commitMap → resolveForeignCommitID → probeCommitID via commitUUIDInverse), so that a 32-char buf-issued UUID carried in the request reference is resolved to its 40-char git SHA BEFORE being passed to `GetFiles`/`GetTree`; then re-run the Phase 21 `TestGenerateWithPinnedBufLock/v1.30.1` e2e regression gate to confirm the v1 protocol works end-to-end.
+**Requirements**: PR-22-1, PR-22-2, PR-22-3, PR-22-4, PR-22-5 (derived from RESEARCH.md Test Map; see 22-01-PLAN.md)
+**Depends on:** Phase 21
+**Plans:** 1/1 plans complete
+Plans:
+
+- [ ] [22-01](./phases/22-fix-v1-30-1-v1alpha1-read-path-uuid-handling-verify-v1-proto/22-01-PLAN.md) — Wire v1alpha1 DownloadManifestAndBlobs to Phase 18 UUID-resolution (CommitResolver interface + resolveCommitForRead wrapper + isUUID branch in blobs.go) with TDD unit cover for PR-22-1/2/3 and the Phase 21 e2e gate for PR-22-4
+
 ---
 
-*Roadmap last updated: 2026-07-07*
+*Roadmap last updated: 2026-07-08*

.planning/STATE.md

@@ -2,15 +2,15 @@
 gsd_state_version: 1.0
 milestone: v1.3
 milestone_name: Diagnostic Logging — In Progress
-status: executing
-last_updated: "2026-07-07T14:17:36.053Z"
-last_activity: 2026-07-07 -- Phase 18 planning complete
+status: "Phase 22 shipped — PR #39"
+last_updated: "2026-07-08T13:36:01.102Z"
+last_activity: 2026-07-08
 progress:
-  total_phases: 8
-  completed_phases: 8
-  total_plans: 10
-  completed_plans: 10
-  percent: 100
+  total_phases: 12
+  completed_phases: 11
+  total_plans: 15
+  completed_plans: 15
+  percent: 92
 ---
 
 # Project State
@@ -25,18 +25,18 @@ See: .planning/PROJECT.md (updated 2026-05-10)
 
 ## Current Position
 
-Phase: 18
-Plan: 18-01 shipped
-Status: Phase 18 plan 01 complete
-Last activity: 2026-07-07 -- Phase 18 plan 01 execution complete (refs, isSHA, prewarm removal, commitUUIDInverse)
+Phase: 22
+Plan: Not started
+Status: Phase 22 shipped — PR #39
+Last activity: 2026-07-08
 
-Progress: [####################] 100%
+Progress: [██████████] 100%
 
 ## Performance Metrics
 
 **Velocity:**
 
-- Total plans completed: 5 (this milestone)
+- Total plans completed: 6 (this milestone)
 - Average duration: ~10 min
 - Total execution time: ~50 min
 
@@ -47,6 +47,7 @@ Progress: [####################] 100%
 | 16    | 3     | -     | -        |
 | 17    | 1     | -     | 8 min    |
 | 18    | 1     | -     | 18 min   |
+| 22 | 1 | - | - |
 
 **Recent Trend:**
 
@@ -79,6 +80,13 @@ None yet.
 - Phase 17 added: Fix PR #37 review findings — address pre-merge issues from Phase 16 PR: re-route digest errors through `logHandlerError`/`upstreamError` (not `internalError`), remove `internalError` helper that bypassed `ERR-05`, accept SHA-256 Bitbucket commits (regression at commits_helpers.go:39), move `preResolveForTest` to a `_test.go` file
 - Phase 18 added: respect buf.yaml dependency refs (not always HEAD); fix related bug; remove prewarm logic now that buf id → git id is derivable
 - Phase 18 planned: 1 plan (18-01-PLAN.md) with 3 tasks — (1) honor `Name.ref` end-to-end + add `isSHA`/`isUUID`/`commitUUIDInverse` helpers + provider ref-resolution; (2) add `commitUUIDInverse` + table-driven tests; (3) delete `prewarmHeads`/`registerResolved`/`PrewarmConfig` and rewrite `probeCommitID` to use the inverse for 32-char UUID inputs. See `.planning/phases/18-respect-buf-yaml-dependency-refs-not-always-head-fix-related/18-{RESEARCH,01-PLAN,VALIDATION}.md`
+- Phase 19 added: e2e tests for the `ref` specified for dependency — `looks like it does not work`. Phase 18 introduced end-to-end honoring of `Name.ref` in `buf.yaml` dependencies (plus `commitUUIDInverse` for 32-char UUID inputs); need a real-server e2e test that exercises the path where the `ref` resolves through buf cache/git to prove it works (or surfaces the bug). See `e2e/ref_test.go` and `.planning/phases/19-we-need-e2e-tests-for-the-ref-specified-for-dependency-looks/`.
+- Phase 19 executed: e2e tests committed in `e15927b feat(19-01): add e2e tests for ref-honoring in buf.yaml deps`. Tests pass token-less (skip cleanly). With `EASYP_GH_TOKEN` set, tests revealed 2 real proxy regressions: (a) v1.30.1 no-ref path tries to resolve `"main"` as a ref → GitHub 422 (also breaks the pre-existing `TestSmokeBufModUpdate`); (b) v1.69.0 ref-pinned run returns HEAD's SHA instead of the ref's SHA on both `buf mod update` and `buf dep update`. The e2e tests successfully caught the regression; the proxy fixes are deferred to Phase 20.
+- Phase 20 added: Fix ref-honoring regressions discovered in Phase 19 — restore the no-ref (HEAD) path for buf v1.30.1 and fix the v1beta1 path where the ref is being ignored on `buf dep update`. See `.planning/phases/20-fix-the-problem-with-refs-discovered-on-phase-19/`.
+- Phase 21 added: e2e test for `buf generate` with `buf.lock` pointing to a valid-but-not-latest commit. This is a valid situation (the lock is intentionally pinned to an older commit) and should work with both v1 and v2 buf CLIs. See `.planning/phases/21-we-need-another-e2e-test-we-are-doing-buf-generate-with-buf-`.
+- Phase 21 executed: `TestGenerateWithPinnedBufLock` committed in `8df1f54 test(21-01): add TestGenerateWithPinnedBufLock matrix test`. With `EASYP_GH_TOKEN` set, the test caught two issues: (a) a deprecated `remote:` field in the plan's buf.gen.yaml — fixed in `e0c79b1 fix(21-01): use 'plugin:' not 'remote:' in generated buf.gen.yaml` (the alpha-remote-generation API was removed in v1.69.0+ and is deprecated in v1.30.1); (b) a real proxy regression in the v1.30.1 v1alpha1 `DownloadManifestAndBlobs` read-path: the proxy passes 32-char buf UUIDs directly to GitHub's `/git/trees/<id>` API which 404s, because the v1alpha1 `Download` chain does not apply `commitUUIDInverse` (only the v1beta1 path from Phase 18 does). The v1.69.0 subtest hits a persistent TLS handshake timeout fetching HEAD's tree from `raw.githubusercontent.com` (and is not actually testing the pinned-UUID path — the v1.69.0 client ignores the `buf.lock` for `buf generate` and just asks the proxy for HEAD). The v1.30.1 fix is deferred to a follow-up phase (proposed `22-fix-v1alpha1-download-uuid-handling`); the v1.69.0 subtest redesign is deferred to another follow-up.
+- Phase 22 proposed: Fix the v1.30.1 v1alpha1 read-path to apply `commitUUIDInverse` on 32-char buf-issued UUIDs in the `DownloadManifestAndBlobs` handler chain, then re-run `TestGenerateWithPinnedBufLock` to confirm both subtests pass. This is the same class of bug Phase 18 fixed for the v1beta1 path; Phase 19/20 e2e tests only exercised the v1beta1 path.
+- Phase 22 added: Fix v1.30.1 v1alpha1 read-path UUID handling; verify v1 protocol works. Depends on Phase 21. Scope: (1) apply `commitUUIDInverse` in the v1alpha1 `DownloadManifestAndBlobs` handler chain so 32-char buf UUIDs resolve to git SHAs before hitting GitHub's tree API; (2) re-run `TestGenerateWithPinnedBufLock` to confirm both subtests pass; (3) broader verification that the v1 (v1alpha1) protocol path works end-to-end (not just `buf generate` — also `buf mod update` and the existing smoke test that Phase 19 revealed was broken for v1.30.1). See `.planning/phases/22-fix-v1-30-1-v1alpha1-read-path-uuid-handling-verify-v1-proto/`.
 
 ## Deferred Items
 
@@ -92,6 +100,6 @@ Items acknowledged and carried forward from previous milestone close:
 
 ## Session Continuity
 
-Last session: 2026-07-07T14:17:36Z
-Stopped at: Phase 18 shipped (1/1 plan, 3 tasks, 5 SCs satisfied, executor self-check PASSED, 18 min)
-Resume file: `.planning/phases/18-respect-buf-yaml-dependency-refs-not-always-head-fix-related/18-01-SUMMARY.md`
+Last session: 2026-07-08T10:47:11Z
+Stopped at: Phase 21 executed (5 commits: tasks 1-2 + initial summary + buf.gen.yaml fix + live-test findings update). Status: verifying. Live test with EASYP_GH_TOKEN caught a real proxy regression in v1.30.1 v1alpha1 read-path (UUID passed to GitHub tree API without commitUUIDInverse). Proposed follow-up Phase 22 to fix the v1alpha1 Download chain. Phase 22 not yet added to ROADMAP.
+Resume file: `.planning/phases/21-we-need-another-e2e-test-we-are-doing-buf-generate-with-buf-/21-01-SUMMARY.md`

.planning/phases/19-we-need-e2e-tests-for-the-ref-specified-for-dependency-looks/.gitkeep

@@ -0,0 +1 @@
+