fix(cli): classify vendor 401s as credential skips in live dogfood

[npwalker]

Intent

The live-dogfood runner scores a command that failed purely because the runner has no credentials as a skip (unavailable for runner credentials), not a fail. That conversion was gated on the 401 body matching one of five hardcoded phrases. Vendor 401 bodies are unstandardized, so an API answering {"code":124,"message":"Invalid access token."} matched nothing and 44 correct no-credential outcomes were scored as http_4xx failures.

Issue: Closes #3827

Approach

Trust the printed CLI's own typed exit code as the primary signal instead of guessing from prose. Printed CLIs return exit 4 from authErr for auth failures, so exit 4 + an HTTP 401 in the output is a credential-unavailable skip regardless of how the vendor worded the body.

The phrase list is kept as the fallback for runs that surface a 401 but exit with a generic failure code, and it gains the wordings observed in the wild (invalid access token, invalid token, expired token, token expired, unauthorized). It moved to a package-level slice consumed by the existing containsAnyOf helper, matching the shape of the neighbouring liveDogfoodRequiredParamFixturePhrases.

Both signals remain gated on http 401 being present, so a 404 whose body happens to contain auth-ish wording is still a genuine failure.

Scope

Primary area: internal/pipeline (live dogfood result classification)

Why this belongs in this repo: this is the Printing Press's own dogfood scorer, not a printed CLI. Every printed CLI's live-dogfood matrix is misscored by the phrase gate, and the fix generalizes across APIs by keying on the generator-emitted typed exit code rather than on any one vendor's message.

Risk

Slightly widens what counts as a credential skip. The widening is bounded by the http 401 precondition, so a real 401 that a runner with valid credentials would still hit could now be skipped instead of failed — an acceptable trade against the current behavior of reporting dozens of false regressions. No generated output, MCP surface, auth, publish, or release behavior changes.

Output Contract

N/A — no template, generated-file, manifest, MCP-schema, scorecard, or pipeline-artifact shape changes. scripts/golden.sh verify was run anyway and passed 32/32 unchanged.

Verification

• go test ./... — every package green. internal/generator (untouched by this change) needs more than the local default deadline on this machine and was re-run alone: go test ./internal/generator/ -timeout 40m -> ok ... 1734s. CI shards that package with its own 12-15m budget.

• scripts/golden.sh verify — pass, 32 cases, no diff

• golangci-lint run ./internal/pipeline/ — 0 issues

• New table-driven test TestLiveDogfoodAuth401TypedExitCode covers the typed-exit-4 path (recognized and unrecognized wording, 401 present and absent, and the generic-exit-code fallback), asserting both liveDogfoodAuth401 and liveDogfoodUnavailableForRunner.

• TestLiveDogfoodAuth401OutputMatchesGooglePhrases extended with the reported vendor shape, a bare Unauthorized body, and a negative case proving a 404 carrying auth wording still fails.

• TestLiveDogfoodUnavailableForRunnerDoesNotHideNotFound extended with the reported vendor shape.

• Generator/template change: verified generated output, including emitted-code assertions or compiled generated CLI output

• Generator/template change: covered the affected fallback or variant shape, not only happy-path fixtures

• Generator/template change: checked emitted definitions and call sites for matching gates

(Not a generator/template change — no emitted Go is affected.)

AI / Automation Disclosure

• No AI or automation was used
• Human-reviewed: AI or automation was used, and a human reviewed the work for intent, fit, and obvious issues before submission
• AI-reviewed only: an AI agent reviewed the work, but no human reviewed it before submission
• Fully automated: generated and submitted without human review for this specific change

🤖 Generated with Claude Code

https://claude.ai/code/session_01BRGzpiy4LuvcAF5QG6dYJc

[mergify]

Merge Protections

🔴 2 of 2 protections blocking · waiting on 🙋 you

	Protection	Waiting on
🔴	require-ready-label-and-ci	🙋 you
🔴	🚦 Auto-queue	🙋 you

🔴 require-ready-label-and-ci

Waiting for any of

• label = ready-to-merge
• title ~= ^chore\(main\): release

This rule is failing.

• any of:
  • label = ready-to-merge
  • all of:
    • head = release-please--branches--main
    • title ~= ^chore\(main\): release
• #changes-requested-reviews-by = 0
• #review-threads-unresolved = 0
• check-success = build-and-test
• check-success = generated-test
• check-success = go-lint
• check-success = golden
• check-success = pr-title
• check-success = test
• any of:
  • -files ~= ^(\.github/workflows/|\.github/scripts/|scripts/|\.github/CODEOWNERS$)
  • approved-reviews-by = mvanhorn
  • approved-reviews-by = tmchow
  • author = mvanhorn
  • author = tmchow
• any of:
  • check-success = Greptile Review
  • check-neutral = Greptile Review
  • check-skipped = Greptile Review
  • head ~= ^mergify/merge-queue/
  • label = queued
  • all of:
    • head = release-please--branches--main
    • title ~= ^chore\(main\): release

🔴 🚦 Auto-queue

Waiting for

• label = ready-to-merge

This rule is failing.

When all merge protections are satisfied and these conditions match, this pull request will be queued automatically.

• label = ready-to-merge

[greptile-apps]

Greptile Summary

This PR improves live-dogfood credential-failure classification.

• Treats printed CLI exit code 4 combined with HTTP 401 output as an unavailable-runner credential skip.
• Expands fallback matching for common vendor token-error wording.
• Adds table-driven coverage for typed exits, generic exits, vendor messages, and non-401 safeguards.

Confidence Score: 5/5

The PR appears safe to merge with no actionable defects identified.

The new classification remains gated on HTTP 401 output, and generated CLIs reserve exit code 4 for auth-classified errors; tests cover both the widened path and its key negative cases.

Important Files Changed

Filename	Overview
internal/pipeline/live_dogfood.go	Uses the generated CLI’s auth-specific exit code alongside the existing HTTP 401 gate and retains phrase matching as a fallback.
internal/pipeline/live_dogfood_test.go	Covers recognized and unrecognized 401 bodies, typed and generic exit codes, and negative non-401 cases.

_{Reviews (1): Last reviewed commit: "fix(cli): classify vendor 401s as creden..." | Re-trigger Greptile}