Skip to content

test(e2e): spawn the real CLI binary in tests, add smoke coverage#118

Merged
ohong merged 3 commits into
mainfrom
oh-cli-e2e
May 5, 2026
Merged

test(e2e): spawn the real CLI binary in tests, add smoke coverage#118
ohong merged 3 commits into
mainfrom
oh-cli-e2e

Conversation

@ohong

@ohong ohong commented May 5, 2026

Copy link
Copy Markdown
Owner

Stacked on #117. Rebase target should change to `main` once #117 merges.

Why

Second "honest follow-up" flagged in #115:

`flows/cli-push-flow.test.ts` and `flows/web-import-flow.test.ts` claim to be flow tests but mock everything. Honest fix is Playwright against a real Next.js + local Supabase. Separate PR.

This PR ships the infrastructure for spawning the real CLI binary in tests, plus a smoke-test exemplar. The full push-flow e2e (which exercises login → real API call → real DB row written) needs either ccusage stubbing infrastructure or a full local Next.js stack — that's tracked as a follow-up so we ship value incrementally rather than bundling everything into one ten-file PR.

What

  • `packages/cli/tests/e2e/spawn.ts` — helper that spawns the built `dist/index.js` as a separate Node process with a tmpdir `HOME`, captured stdout/stderr/exit, configurable env. The truncate-stdout option exists so future EPIPE tests can drive the same code path users hit when piping to `head`.
  • `packages/cli/tests/e2e/cli-smoke.e2e.test.ts` — six smoke cases:
    • `--help` exits 0 and prints `Usage:` / `straude `
    • `-h` is a real alias
    • `--version` exits 0 and prints `straude v${pkg.version}`
    • `-v` is a real alias
    • Unknown command exits non-zero with `Unknown command` on stderr
    • `--version` is a side-effect-free read (no `~/.straude` directory created)

What this catches that in-process tests cannot

  • Build-pipeline regressions. tsc emits broken JS, a missing import, a stale dist/. Every other CLI test imports from `src/`; this one imports from `dist/`.
  • argv parser drift. The `parseArgs` function is unit-testable, but the wiring between argv → branches in `main()` is not — until you spawn it.
  • Real exit codes. `process.exit(1)` from the help/error path produces an actual non-zero exit code, not a thrown ExitError mock.
  • Real stdout the user reads. Newlines, encoding, terminal-detect behavior — all of it.

Test plan

  • `bun run --cwd packages/cli build` — green
  • `bun run --cwd packages/cli test` — 226 tests, all green (was 220 on main, +6 new)
  • No new CI step needed — the existing `Build` step in `.github/workflows/ci.yml` already produces `dist/` before `Test (cli)` runs

Out of scope (real follow-ups)

  • Full push-flow e2e. Requires either: (a) a stub `ccusage` binary that the test drops on `PATH` and returns canned JSON, or (b) the real `bunx supabase start` + `bun run --cwd apps/web start` + a real `ccusage` install. The first is ~1-2 hours; the second piggybacks on test(integration): real-Supabase API route tests via bunx supabase start #117's infra and is the more honest version. I'll ship one of these as a follow-up.
  • Login flow e2e. Spawn `straude login` against an in-process HTTP listener emulating `/api/auth/cli/init` + `/api/auth/cli/poll`. Easy once we have the spawn helper (this PR).
  • EPIPE regression test. The `truncateStdout` flag in `spawn.ts` is wired specifically for this — `spawn(`straude --help`, { truncateStdout: true })` should exit 0 once fix(cli): unblock activation — ccusage install, EPIPE, silent reauth, backfill filter #114's EPIPE handler lands.

🤖 Generated with Claude Code

Foundation for the second "honest follow-up" flagged in #115: replace the
fully mocked `flows/cli-push-flow.test.ts` with tests that exercise the
real binary. This PR ships the spawn-helper infrastructure plus six smoke
tests; full push-flow e2e (which needs ccusage stubbing or a real local
stack) is deferred to a follow-up.

What's new:
- packages/cli/__tests__/e2e/spawn.ts — helper that spawns
  `node dist/index.js` with a tmpdir HOME, controlled env, and captured
  stdout/stderr/exit. No mocks of fs, env, or fetch.
- packages/cli/__tests__/e2e/cli-smoke.e2e.test.ts — six smoke cases:
  --help / -h alias / --version / -v alias / unknown-command exit /
  --version is a side-effect-free read.

What this catches that in-process tests cannot:
- Build-pipeline regressions (tsc emits broken JS, missing imports).
- argv parsing breakage that doesn't surface when calling main() directly.
- Real exit codes for help/version/error paths.
- Real stdout output the user reads.

Test count: 220 → 226. CI runs the e2e tests as part of the existing
`bun run --cwd packages/cli test` step (no new step needed; the build
step that runs before tests has always built dist/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented May 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
straude Ready Ready Preview, Comment May 5, 2026 9:32pm

Request Review

@coderabbitai

coderabbitai Bot commented May 5, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@ohong has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 33 minutes before requesting another review.

To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9d0a46fc-aa9c-4657-b6a3-f2144963af9b

📥 Commits

Reviewing files that changed from the base of the PR and between 482320d and 5b7e050.

📒 Files selected for processing (3)
  • docs/CHANGELOG.md
  • packages/cli/__tests__/e2e/cli-smoke.e2e.test.ts
  • packages/cli/__tests__/e2e/spawn.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch oh-cli-e2e

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ohong ohong changed the base branch from oh-integration-tests-postgres to main May 5, 2026 21:30
@ohong ohong merged commit f78af30 into main May 5, 2026
5 checks passed
@ohong ohong deleted the oh-cli-e2e branch June 11, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant