flakesage

Find flaky tests by running your suite a few times and diffing the results. Language-agnostic (anything that emits JUnit XML), read-only, and it tells you which tests are unreliable — not just that "something failed."

A flaky test is the worst kind of failure: it passes, you rerun, it passes again, so you shrug and merge — until it fails in someone else's PR and everyone stops trusting the suite. Figuring out which tests are actually flaky is painfully manual: you rerun the whole thing over and over, eyeball what changed, and try to remember which failures were "real."

flakesage does that for you. It runs your test suite several times, compares the results run-to-run, and points at exactly the tests whose outcome flips — with a flake rate and an optional AI guess at the cause.

🎲 1 flaky test(s) across 3 runs (3 tests):

  ● tests.test_mod::test_b  x·x  67% fail
     likely: shared-state — outcome varies between identical runs, usually state leaking between tests.

Why this isn't just "rerun until green"

• A test that fails every run is a real failure — flakesage separates those out so you don't chase them as flakes.
• It works off JUnit XML, which nearly every test runner emits (pytest, jest, go, JUnit, etc.), so it isn't tied to one language or framework.
• A chatbot can't help here — it can't run your suite 50 times. This is the kind of tool that earns its place by doing something a chat window fundamentally can't.

Install

git clone https://github.com/jay-tank/flakesage.git
cd flakesage
pip install .
# optional, for --explain with a hosted model:
pip install '.[claude]'   # or '.[openai]'

Using it

Two ways in. If you already have per-run JUnit reports (say from CI), just point at them:

flakesage --junit run1.xml run2.xml run3.xml

Or let it run your suite N times. Your command just needs to write a JUnit report to the path flakesage gives it:

flakesage --runs 10 --report report.xml -- pytest --junitxml=report.xml

Add --explain to have an LLM classify the likely cause (order-dependence, timing, randomness, shared state…). It's off by default — the detection itself needs no key:

flakesage --junit run*.xml --explain

Exit codes: 0 = no flaky tests, 1 = flaky tests found, 2 = usage error. So it drops into CI as a gate that fails the build when your suite is unreliable.

More in docs/USAGE.md and docs/PROVIDERS.md.

Good to know

• It's read-only — it runs your tests and reads reports, nothing else.
• XML is parsed with defusedxml, so a malformed or hostile report can't XXE you.
• The more runs, the more confident the result. For a quick check 5 is fine; to catch a rare flake, bump --runs up.

Limitations

• It catches flakiness that shows up across repeated identical runs. Order-dependent flakes only surface if your runner shuffles order (e.g. add pytest-randomly to your command) — flakesage will happily analyze those runs too.
• --explain is a best-effort guess to point you in a direction, not a verdict.

License

MIT — see LICENSE.