Skip to content

ci/ai: check out the failure SHA before the /investigate agent runs #172037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rickystewart merged 1 commit into from
Jun 26, 2026
+84 −73
Merged

ci/ai: check out the failure SHA before the /investigate agent runs #172037

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 28 additions & 28 deletions .github/prompts/investigate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,26 +12,27 @@ prompt); `gh` defaults to it, so use plain `gh issue`/`gh pr`/`gh search`
commands. The working tree is checked out from `CODE_REPO`; use that
when building source links (blob/permalink URLs).

You are inside a blobless clone of the `CODE_REPO` repository with
full commit history. `git log`, `git diff`, etc. work normally — no
need to deepen or unshallow.

To inspect a specific commit, check it out normally:

```bash
git checkout <sha>
```

File contents for any reachable commit are fetched on demand, so this
works even for commits on release branches. If the checkout fails
because the SHA is not reachable from this remote (rare — only for
commits exclusive to a fork), proceed with the currently checked-out
code instead, but add a prominent warning at the very top of
You are inside a blobless clone of the `CODE_REPO` repository. `git log`
works across full history. The working tree is **already checked out at
the failure commit** (`CHECKED OUT SHA`, passed in the prompt); reading
files, `git blame`, `git show`, and `git diff` against that commit all
work because its file contents are present locally.

Do **not** run `git checkout`, and do not try to read file contents at a
*different* commit. The agent's git credentials cannot fetch from
`CODE_REPO`, so anything that needs another commit's file contents (a
checkout, `git show <other-sha>:file`, `git blame` walking into older
revisions) will fail. The failure-commit checkout is performed for you in
an earlier workflow step — just analyze the code as checked out.

If `CHECKED OUT SHA` is empty, the failure SHA could not be checked out
and the working tree is at the default-branch tip instead. Proceed with
the available code but add a prominent warning at the very top of
`artifacts/findings.md`:

> **Warning:** Could not check out failure SHA `<sha>`. Analysis is
> based on the default branch tip, which may differ from the code
> that produced the failure.
> **Warning:** Could not check out the failure SHA. Analysis is based on
> the default branch tip, which may differ from the code that produced
> the failure.

Failure types include roachtests (`pkg/cmd/roachtest/tests`) and Go
unit tests. Your goal is to develop hypotheses about the failure and
Expand All @@ -47,7 +48,8 @@ to check what's available.
Key tools at your disposal:

- **Code reading**: Read, Grep, Glob, and common shell text tools
- **Git**: all git commands (log, diff, show, checkout, etc.)
- **Git**: read-only git commands (log, diff, show, blame) against the
checked-out failure commit; not `git checkout` (see above)
- **GitHub CLI**: gh issue view/list, gh pr view/list/diff, gh search
- **Web browsing**: WebFetch tool for reading web pages and JSON APIs
- **File download**: `fetch-url <url> [output-file]` (GET-only HTTP
Expand Down Expand Up @@ -125,16 +127,14 @@ fix is present in the failure SHA's history using `git log`.

### Step 3: Read the Source Code

After determining the failure SHA from Step 1, check it out:

```bash
git checkout <failure-sha>
```

If the checkout fails, fall back to the default branch tip (see the
warning note in the checkout instructions above).
The working tree is already checked out at the failure commit
(`CHECKED OUT SHA`), so explore the source directly. Confirm that
`CHECKED OUT SHA` matches the failure you are investigating from Step 1;
if it does not (or it is empty), note that the analysis is pinned to the
checked-out code — you cannot switch commits (see the checkout
instructions above).

Then explore the relevant source code:
Explore the relevant source code:
- Roachtest code lives in `pkg/cmd/roachtest/tests/`
- Unit tests live alongside their package
- Grep for error messages to find their origin
Expand Down
101 changes: 56 additions & 45 deletions .github/workflows/investigate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,13 @@ jobs:
# haves, which GitHub serves reliably for any reachable commit — and
# import those objects so later reads need no network access.
#
# All git invocations here use the real binary (/usr/bin/git), not
# the wrapper installed below, to avoid recursion.
# This runs in a workflow step (not inside the agent), while the
# actions/checkout credentials are still in place. The agent step
# never has working credentials for CODE_REPO: cockroachdb/claude-
# code-action's configureGitAuth strips the actions/checkout auth
# header and repoints origin at the workflow repo, so any git fetch
# the agent attempts hits the wrong repository. That is why the
# checkout is done here rather than by the agent.
cat > /usr/local/bin/snapshot-fetch <<'WRAPPER'
#!/bin/bash
set -euo pipefail
Expand Down Expand Up @@ -173,59 +178,59 @@ jobs:
set -euo pipefail
sha="${1:?Usage: checkout-sha SHA}"
snapshot-fetch "$sha"
/usr/bin/git -c advice.detachedHead=false checkout --force "$sha"
git -c advice.detachedHead=false checkout --force "$sha"
WRAPPER
chmod +x /usr/local/bin/checkout-sha

# git wrapper — make a plain `git checkout <sha>` (the reflex the
# agent reaches for) work in the blobless clone. Real git handles
# everything; only a checkout/switch that fails because the lazy
# blob fetch was rejected ("not our ref" / promisor remote) is
# retried after snapshot-fetch imports the target commit. This
# removes the dependency on the agent remembering to use
# checkout-sha, which it reliably forgets.
cat > /usr/local/bin/git <<'WRAPPER'
#!/bin/bash
# Pin the worktree to the failure commit while the actions/checkout
# credentials for the checked-out repo are still active. The agent step
# cannot do this itself: cockroachdb/claude-code-action's configureGitAuth
# strips that auth header and repoints origin at the workflow repo, so any
# fetch the agent attempts (e.g. a lazy blob fetch on `git checkout`) hits
# the wrong repository and fails with "not our ref". We therefore resolve
# and check out the SHA here and pass it to the agent. On any failure we
# leave the worktree at the default-branch tip; the agent emits a warning.
- name: Check out failure SHA
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -uo pipefail
REAL_GIT=/usr/bin/git

case "${1:-}" in
checkout|switch) ;;
*) exec "$REAL_GIT" "$@" ;;
esac

out=$("$REAL_GIT" "$@" 2>&1); rc=$?
if [ "$rc" -eq 0 ]; then
[ -n "$out" ] && printf '%s\n' "$out" >&2
exit 0
# An explicit SHA in the trigger comment ("/investigate <sha>") wins.
sha=""
if [[ "$COMMENT_BODY" =~ /investigate[[:space:]]+([0-9a-f]{40}) ]]; then
sha="${BASH_REMATCH[1]}"
fi
if ! printf '%s' "$out" | grep -qiE 'promisor|not our ref'; then
printf '%s\n' "$out" >&2
exit "$rc"

# Otherwise take the most recent failure SHA. Failures are filed by
# cockroach-teamcity (the issue body for the first, a comment for each
# subsequent one), each linking to .../commits/<sha>. Prefer the
# latest cockroach-teamcity post; fall back to the latest commit link
# from any author.
if [ -z "$sha" ]; then
sha=$(gh issue view "$ISSUE_NUMBER" --repo "${{ github.repository }}" \
--json author,body,comments --jq '
([{a: .author.login, b: .body}] + [.comments[] | {a: .author.login, b: .body}]) as $posts
| ([ $posts[] | select(.a=="cockroach-teamcity") | .b | [scan("commits?/([0-9a-f]{40})")] | (.[0][0] // empty) ]) as $tc
| ([ $posts[] | .b | [scan("commits?/([0-9a-f]{40})")] | (.[0][0] // empty) ]) as $any
| (if ($tc|length)>0 then ($tc|last) else ($any|last // empty) end)
' 2>/dev/null) || sha=""
fi

# Find the commit-ish among the arguments: the first non-option
# token before a `--` path separator that resolves to a commit.
# The full commit graph is present (blobless clone), so rev-parse
# resolves offline.
sha=""
for arg in "${@:2}"; do
[ "$arg" = "--" ] && break
case "$arg" in -*) continue ;; esac
if "$REAL_GIT" rev-parse --verify --quiet "${arg}^{commit}" >/dev/null 2>&1; then
sha=$("$REAL_GIT" rev-parse "${arg}^{commit}")
break
fi
done
if [ -z "$sha" ]; then
printf '%s\n' "$out" >&2
exit "$rc"
echo "::warning::Could not determine a failure SHA from issue $ISSUE_NUMBER; leaving worktree at default-branch tip."
echo "FAILURE_SHA=" >> "$GITHUB_ENV"
exit 0
fi

snapshot-fetch "$sha"
exec "$REAL_GIT" "$@"
WRAPPER
chmod +x /usr/local/bin/git
echo "Determined failure SHA: $sha"
if checkout-sha "$sha"; then
echo "Checked out $sha"
echo "FAILURE_SHA=$sha" >> "$GITHUB_ENV"
else
echo "::warning::Failed to check out failure SHA $sha; leaving worktree at default-branch tip."
echo "FAILURE_SHA=" >> "$GITHUB_ENV"
fi

# Vertex AI auth for cockroachdb/cockroach. Skipped when an
# ANTHROPIC_API_KEY secret is set (e.g. on a personal fork).
Expand Down Expand Up @@ -274,7 +279,7 @@ jobs:
# based permissions may work after upgrading the action.
claude_args: |
--model ${{ inputs.cheap == true && 'claude-sonnet-4-5' || 'claude-opus-4-6' }}
--allowedTools "Write,Read,Grep,Glob,WebFetch,Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(rg:*),Bash(awk:*),Bash(cut:*),Bash(tr:*),Bash(sort:*),Bash(uniq:*),Bash(wc:*),Bash(tee:*),Bash(diff:*),Bash(file:*),Bash(strings:*),Bash(jq:*),Bash(ls:*),Bash(find:*),Bash(tree:*),Bash(stat:*),Bash(du:*),Bash(mkdir:*),Bash(git:*),Bash(checkout-sha:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh pr diff:*),Bash(gh search:*),Bash(fetch-url:*),Bash(unzip:*),Bash(tar x*),Bash(tar -x*),Bash(tar --extract:*),Bash(go mod download:*),Bash(go env:*),Bash(.claude/skills/engflow-artifacts/run.sh:*),Bash(go tool pprof:*),Bash(go run ./pkg/cmd/tsdump2duck:*),Bash(duckdb:*)"
--allowedTools "Write,Read,Grep,Glob,WebFetch,Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(rg:*),Bash(awk:*),Bash(cut:*),Bash(tr:*),Bash(sort:*),Bash(uniq:*),Bash(wc:*),Bash(tee:*),Bash(diff:*),Bash(file:*),Bash(strings:*),Bash(jq:*),Bash(ls:*),Bash(find:*),Bash(tree:*),Bash(stat:*),Bash(du:*),Bash(mkdir:*),Bash(git:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh pr diff:*),Bash(gh search:*),Bash(fetch-url:*),Bash(unzip:*),Bash(tar x*),Bash(tar -x*),Bash(tar --extract:*),Bash(go mod download:*),Bash(go env:*),Bash(.claude/skills/engflow-artifacts/run.sh:*),Bash(go tool pprof:*),Bash(go run ./pkg/cmd/tsdump2duck:*),Bash(duckdb:*)"
prompt: |
Read and follow the instructions in the prompt file
`.github/prompts/${{ inputs.smoke_test == true && 'investigate-smoke' || 'investigate' }}.md`.
Expand All @@ -283,11 +288,17 @@ jobs:
CODE_REPO: ${{ env.HAS_API_KEY == 'true' && github.repository || env.CODE_REPO }}
ISSUE NUMBER: ${{ env.ISSUE_NUMBER }}
TRIGGER COMMENT: ${{ env.COMMENT_BODY }}
CHECKED OUT SHA: ${{ env.FAILURE_SHA }}
WORKFLOW RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

Use ISSUE_REPO for all gh issue/pr/search commands. Use
CODE_REPO when building source links (blob/permalink URLs).

The working tree is already checked out at CHECKED OUT SHA (the
failure commit). If CHECKED OUT SHA is empty, the working tree is
at the default-branch tip instead. Do not run `git checkout` — the
agent's git credentials cannot fetch from CODE_REPO.

- name: Upload findings
if: always()
uses: actions/upload-artifact@v4
Expand Down
Loading