From 4b86a0aa5d3796f60cfdc3073da9189d9c5e6abb Mon Sep 17 00:00:00 2001 From: Ricky Stewart Date: Fri, 26 Jun 2026 14:17:10 -0500 Subject: [PATCH] ci/ai: check out the failure SHA before the /investigate agent runs The /investigate agent could not check out the failure commit. Its `git checkout ` failed with "not our ref", and it fell back to analyzing the default-branch tip -- the persistent problem reported for the workflow. Prior fixes (the checkout-sha helper, then a git wrapper that retried via snapshot-fetch) assumed GitHub's promisor rejects object-level fetches for commits off the default branch. That was a misdiagnosis: those failures were all observed inside the agent step, where the credentials were already broken. The real cause is in the action that runs the agent: in agent mode it calls configureGitAuth during prepare, which removes the auth header actions/checkout installed for CODE_REPO and repoints origin at the workflow repo using the default GITHUB_TOKEN. By the time the agent runs, origin no longer points at CODE_REPO and the credential is gone, so any fetch the agent attempts -- including the lazy blob fetch a checkout triggers -- hits the wrong repository and fails. Stop relying on the agent's git for this. Resolve and check out the failure SHA in a dedicated workflow step that runs while the actions/checkout credentials are still active, then pass the SHA to the agent and tell it the tree is already checked out. The SHA is taken from an explicit `/investigate ` trigger, otherwise the most recent failure linked in the issue. On any failure the worktree is left at the default-branch tip and the agent emits the existing warning. This lets the git wrapper be removed (snapshot-fetch and checkout-sha remain, now used by the workflow step). It also keeps INVESTIGATE_PAT away from the agent: configureGitAuth still strips it before the agent runs, so the token is never exposed to it. Epic: none Release note: none --- .github/prompts/investigate.md | 56 ++++++++--------- .github/workflows/investigate.yml | 101 +++++++++++++++++------------- 2 files changed, 84 insertions(+), 73 deletions(-) diff --git a/.github/prompts/investigate.md b/.github/prompts/investigate.md index 7d105291a8cd..546cf5a70185 100644 --- a/.github/prompts/investigate.md +++ b/.github/prompts/investigate.md @@ -12,26 +12,27 @@ prompt); `gh` defaults to it, so use plain `gh issue`/`gh pr`/`gh search` commands. The working tree is checked out from `CODE_REPO`; use that when building source links (blob/permalink URLs). -You are inside a blobless clone of the `CODE_REPO` repository with -full commit history. `git log`, `git diff`, etc. work normally — no -need to deepen or unshallow. - -To inspect a specific commit, check it out normally: - -```bash -git checkout -``` - -File contents for any reachable commit are fetched on demand, so this -works even for commits on release branches. If the checkout fails -because the SHA is not reachable from this remote (rare — only for -commits exclusive to a fork), proceed with the currently checked-out -code instead, but add a prominent warning at the very top of +You are inside a blobless clone of the `CODE_REPO` repository. `git log` +works across full history. The working tree is **already checked out at +the failure commit** (`CHECKED OUT SHA`, passed in the prompt); reading +files, `git blame`, `git show`, and `git diff` against that commit all +work because its file contents are present locally. + +Do **not** run `git checkout`, and do not try to read file contents at a +*different* commit. The agent's git credentials cannot fetch from +`CODE_REPO`, so anything that needs another commit's file contents (a +checkout, `git show :file`, `git blame` walking into older +revisions) will fail. The failure-commit checkout is performed for you in +an earlier workflow step — just analyze the code as checked out. + +If `CHECKED OUT SHA` is empty, the failure SHA could not be checked out +and the working tree is at the default-branch tip instead. Proceed with +the available code but add a prominent warning at the very top of `artifacts/findings.md`: -> **Warning:** Could not check out failure SHA ``. Analysis is -> based on the default branch tip, which may differ from the code -> that produced the failure. +> **Warning:** Could not check out the failure SHA. Analysis is based on +> the default branch tip, which may differ from the code that produced +> the failure. Failure types include roachtests (`pkg/cmd/roachtest/tests`) and Go unit tests. Your goal is to develop hypotheses about the failure and @@ -47,7 +48,8 @@ to check what's available. Key tools at your disposal: - **Code reading**: Read, Grep, Glob, and common shell text tools -- **Git**: all git commands (log, diff, show, checkout, etc.) +- **Git**: read-only git commands (log, diff, show, blame) against the + checked-out failure commit; not `git checkout` (see above) - **GitHub CLI**: gh issue view/list, gh pr view/list/diff, gh search - **Web browsing**: WebFetch tool for reading web pages and JSON APIs - **File download**: `fetch-url [output-file]` (GET-only HTTP @@ -125,16 +127,14 @@ fix is present in the failure SHA's history using `git log`. ### Step 3: Read the Source Code -After determining the failure SHA from Step 1, check it out: - -```bash -git checkout -``` - -If the checkout fails, fall back to the default branch tip (see the -warning note in the checkout instructions above). +The working tree is already checked out at the failure commit +(`CHECKED OUT SHA`), so explore the source directly. Confirm that +`CHECKED OUT SHA` matches the failure you are investigating from Step 1; +if it does not (or it is empty), note that the analysis is pinned to the +checked-out code — you cannot switch commits (see the checkout +instructions above). -Then explore the relevant source code: +Explore the relevant source code: - Roachtest code lives in `pkg/cmd/roachtest/tests/` - Unit tests live alongside their package - Grep for error messages to find their origin diff --git a/.github/workflows/investigate.yml b/.github/workflows/investigate.yml index eac007c811b0..63dbd4bc258b 100644 --- a/.github/workflows/investigate.yml +++ b/.github/workflows/investigate.yml @@ -141,8 +141,13 @@ jobs: # haves, which GitHub serves reliably for any reachable commit — and # import those objects so later reads need no network access. # - # All git invocations here use the real binary (/usr/bin/git), not - # the wrapper installed below, to avoid recursion. + # This runs in a workflow step (not inside the agent), while the + # actions/checkout credentials are still in place. The agent step + # never has working credentials for CODE_REPO: cockroachdb/claude- + # code-action's configureGitAuth strips the actions/checkout auth + # header and repoints origin at the workflow repo, so any git fetch + # the agent attempts hits the wrong repository. That is why the + # checkout is done here rather than by the agent. cat > /usr/local/bin/snapshot-fetch <<'WRAPPER' #!/bin/bash set -euo pipefail @@ -173,59 +178,59 @@ jobs: set -euo pipefail sha="${1:?Usage: checkout-sha SHA}" snapshot-fetch "$sha" - /usr/bin/git -c advice.detachedHead=false checkout --force "$sha" + git -c advice.detachedHead=false checkout --force "$sha" WRAPPER chmod +x /usr/local/bin/checkout-sha - # git wrapper — make a plain `git checkout ` (the reflex the - # agent reaches for) work in the blobless clone. Real git handles - # everything; only a checkout/switch that fails because the lazy - # blob fetch was rejected ("not our ref" / promisor remote) is - # retried after snapshot-fetch imports the target commit. This - # removes the dependency on the agent remembering to use - # checkout-sha, which it reliably forgets. - cat > /usr/local/bin/git <<'WRAPPER' - #!/bin/bash + # Pin the worktree to the failure commit while the actions/checkout + # credentials for the checked-out repo are still active. The agent step + # cannot do this itself: cockroachdb/claude-code-action's configureGitAuth + # strips that auth header and repoints origin at the workflow repo, so any + # fetch the agent attempts (e.g. a lazy blob fetch on `git checkout`) hits + # the wrong repository and fails with "not our ref". We therefore resolve + # and check out the SHA here and pass it to the agent. On any failure we + # leave the worktree at the default-branch tip; the agent emits a warning. + - name: Check out failure SHA + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | set -uo pipefail - REAL_GIT=/usr/bin/git - case "${1:-}" in - checkout|switch) ;; - *) exec "$REAL_GIT" "$@" ;; - esac - - out=$("$REAL_GIT" "$@" 2>&1); rc=$? - if [ "$rc" -eq 0 ]; then - [ -n "$out" ] && printf '%s\n' "$out" >&2 - exit 0 + # An explicit SHA in the trigger comment ("/investigate ") wins. + sha="" + if [[ "$COMMENT_BODY" =~ /investigate[[:space:]]+([0-9a-f]{40}) ]]; then + sha="${BASH_REMATCH[1]}" fi - if ! printf '%s' "$out" | grep -qiE 'promisor|not our ref'; then - printf '%s\n' "$out" >&2 - exit "$rc" + + # Otherwise take the most recent failure SHA. Failures are filed by + # cockroach-teamcity (the issue body for the first, a comment for each + # subsequent one), each linking to .../commits/. Prefer the + # latest cockroach-teamcity post; fall back to the latest commit link + # from any author. + if [ -z "$sha" ]; then + sha=$(gh issue view "$ISSUE_NUMBER" --repo "${{ github.repository }}" \ + --json author,body,comments --jq ' + ([{a: .author.login, b: .body}] + [.comments[] | {a: .author.login, b: .body}]) as $posts + | ([ $posts[] | select(.a=="cockroach-teamcity") | .b | [scan("commits?/([0-9a-f]{40})")] | (.[0][0] // empty) ]) as $tc + | ([ $posts[] | .b | [scan("commits?/([0-9a-f]{40})")] | (.[0][0] // empty) ]) as $any + | (if ($tc|length)>0 then ($tc|last) else ($any|last // empty) end) + ' 2>/dev/null) || sha="" fi - # Find the commit-ish among the arguments: the first non-option - # token before a `--` path separator that resolves to a commit. - # The full commit graph is present (blobless clone), so rev-parse - # resolves offline. - sha="" - for arg in "${@:2}"; do - [ "$arg" = "--" ] && break - case "$arg" in -*) continue ;; esac - if "$REAL_GIT" rev-parse --verify --quiet "${arg}^{commit}" >/dev/null 2>&1; then - sha=$("$REAL_GIT" rev-parse "${arg}^{commit}") - break - fi - done if [ -z "$sha" ]; then - printf '%s\n' "$out" >&2 - exit "$rc" + echo "::warning::Could not determine a failure SHA from issue $ISSUE_NUMBER; leaving worktree at default-branch tip." + echo "FAILURE_SHA=" >> "$GITHUB_ENV" + exit 0 fi - snapshot-fetch "$sha" - exec "$REAL_GIT" "$@" - WRAPPER - chmod +x /usr/local/bin/git + echo "Determined failure SHA: $sha" + if checkout-sha "$sha"; then + echo "Checked out $sha" + echo "FAILURE_SHA=$sha" >> "$GITHUB_ENV" + else + echo "::warning::Failed to check out failure SHA $sha; leaving worktree at default-branch tip." + echo "FAILURE_SHA=" >> "$GITHUB_ENV" + fi # Vertex AI auth for cockroachdb/cockroach. Skipped when an # ANTHROPIC_API_KEY secret is set (e.g. on a personal fork). @@ -274,7 +279,7 @@ jobs: # based permissions may work after upgrading the action. claude_args: | --model ${{ inputs.cheap == true && 'claude-sonnet-4-5' || 'claude-opus-4-6' }} - --allowedTools "Write,Read,Grep,Glob,WebFetch,Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(rg:*),Bash(awk:*),Bash(cut:*),Bash(tr:*),Bash(sort:*),Bash(uniq:*),Bash(wc:*),Bash(tee:*),Bash(diff:*),Bash(file:*),Bash(strings:*),Bash(jq:*),Bash(ls:*),Bash(find:*),Bash(tree:*),Bash(stat:*),Bash(du:*),Bash(mkdir:*),Bash(git:*),Bash(checkout-sha:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh pr diff:*),Bash(gh search:*),Bash(fetch-url:*),Bash(unzip:*),Bash(tar x*),Bash(tar -x*),Bash(tar --extract:*),Bash(go mod download:*),Bash(go env:*),Bash(.claude/skills/engflow-artifacts/run.sh:*),Bash(go tool pprof:*),Bash(go run ./pkg/cmd/tsdump2duck:*),Bash(duckdb:*)" + --allowedTools "Write,Read,Grep,Glob,WebFetch,Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(rg:*),Bash(awk:*),Bash(cut:*),Bash(tr:*),Bash(sort:*),Bash(uniq:*),Bash(wc:*),Bash(tee:*),Bash(diff:*),Bash(file:*),Bash(strings:*),Bash(jq:*),Bash(ls:*),Bash(find:*),Bash(tree:*),Bash(stat:*),Bash(du:*),Bash(mkdir:*),Bash(git:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh pr diff:*),Bash(gh search:*),Bash(fetch-url:*),Bash(unzip:*),Bash(tar x*),Bash(tar -x*),Bash(tar --extract:*),Bash(go mod download:*),Bash(go env:*),Bash(.claude/skills/engflow-artifacts/run.sh:*),Bash(go tool pprof:*),Bash(go run ./pkg/cmd/tsdump2duck:*),Bash(duckdb:*)" prompt: | Read and follow the instructions in the prompt file `.github/prompts/${{ inputs.smoke_test == true && 'investigate-smoke' || 'investigate' }}.md`. @@ -283,11 +288,17 @@ jobs: CODE_REPO: ${{ env.HAS_API_KEY == 'true' && github.repository || env.CODE_REPO }} ISSUE NUMBER: ${{ env.ISSUE_NUMBER }} TRIGGER COMMENT: ${{ env.COMMENT_BODY }} + CHECKED OUT SHA: ${{ env.FAILURE_SHA }} WORKFLOW RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} Use ISSUE_REPO for all gh issue/pr/search commands. Use CODE_REPO when building source links (blob/permalink URLs). + The working tree is already checked out at CHECKED OUT SHA (the + failure commit). If CHECKED OUT SHA is empty, the working tree is + at the default-branch tip instead. Do not run `git checkout` — the + agent's git credentials cannot fetch from CODE_REPO. + - name: Upload findings if: always() uses: actions/upload-artifact@v4