Skip to content

Rewrite assertions before test execution - #1109

Open
MatthewMckee4 wants to merge 1 commit into
mainfrom
codex/issue-1082
Open

Rewrite assertions before test execution#1109
MatthewMckee4 wants to merge 1 commit into
mainfrom
codex/issue-1082

Conversation

@MatthewMckee4

@MatthewMckee4 MatthewMckee4 commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Summary

Rewrite assertions while importing collected test modules so message-less AssertionError values include the evaluated expression. Fixes #1082.

Test Plan

cargo insta test -p karva --accept --test-runner nextest and uvx prek run -a.

@MatthewMckee4 MatthewMckee4 added the test-running Related to running tests with karva label Aug 1, 2026
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

❌ Merging this PR may alter performance

Baseline: main (ce8ff7d). Candidate: PR (59f0d35). Each benchmark compares median CLI wall time on one GitHub Actions runner, alternating install order. Runs warm the duration cache before measuring and include default per-test status output. Lower is better.

0 improved benchmarks
13 regressed benchmarks
1 unchanged benchmark

Warning

Benchmark regressions were detected. Review the wall-time changes before merging.

Performance Changes

Mode Benchmark Base Head Change Runs
WallTime requests 16.040 s 16.882 s +5.3% 4
WallTime fastapi 6.768 s 8.588 s +26.9% 7
WallTime httpx 9.137 s 9.781 s +7.1% 7
WallTime h11 395.1 ms 566.6 ms +43.4% 21
WallTime markupsafe 252.8 ms 272.5 ms +7.8% 21
WallTime itsdangerous 268.6 ms 289.0 ms +7.6% 21
WallTime blinker 287.6 ms 308.4 ms +7.2% 21
WallTime jinja 574.5 ms 796.7 ms +38.7% 21
WallTime installer 382.1 ms 403.0 ms +5.5% 21
WallTime tomlkit 655.6 ms 940.5 ms +43.5% 15
WallTime outcome 281.2 ms 301.7 ms +7.3% 21
WallTime pluggy 333.6 ms 483.8 ms +45.1% 21
WallTime werkzeug 9.039 s 10.267 s +13.6% 7
All benchmark scores
Mode Benchmark Base Head Change Runs
WallTime requests 16.040 s 16.882 s +5.3% 4
WallTime fastapi 6.768 s 8.588 s +26.9% 7
WallTime httpx 9.137 s 9.781 s +7.1% 7
WallTime h11 395.1 ms 566.6 ms +43.4% 21
WallTime markupsafe 252.8 ms 272.5 ms +7.8% 21
WallTime sniffio 269.1 ms 268.5 ms -0.2% 21
WallTime itsdangerous 268.6 ms 289.0 ms +7.6% 21
WallTime blinker 287.6 ms 308.4 ms +7.2% 21
WallTime jinja 574.5 ms 796.7 ms +38.7% 21
WallTime installer 382.1 ms 403.0 ms +5.5% 21
WallTime tomlkit 655.6 ms 940.5 ms +43.5% 15
WallTime outcome 281.2 ms 301.7 ms +7.3% 21
WallTime pluggy 333.6 ms 483.8 ms +45.1% 21
WallTime werkzeug 9.039 s 10.267 s +13.6% 7

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

❌ Merging this PR may increase memory usage

Baseline: main (ce8ff7d). Candidate: PR (59f0d35). Each benchmark compares median peak RSS for the installed Karva CLI on one GitHub Actions runner, alternating install order. Runs warm the duration cache before measuring and are configured per project. Lower is better.

0 improved benchmarks
10 regressed benchmarks
4 unchanged benchmarks

Warning

Benchmark regressions were detected. Review the peak-memory changes before merging.

Performance Changes

Mode Benchmark Base Head Change Runs
Memory requests 60.2 MiB 69.3 MiB +15.0% 4
Memory fastapi 146.1 MiB 148.0 MiB +1.3% 7
Memory httpx 81.0 MiB 81.9 MiB +1.1% 7
Memory h11 53.1 MiB 55.3 MiB +4.0% 21
Memory markupsafe 41.9 MiB 42.9 MiB +2.3% 21
Memory itsdangerous 44.1 MiB 44.8 MiB +1.5% 21
Memory jinja 56.9 MiB 58.3 MiB +2.4% 21
Memory tomlkit 49.1 MiB 50.0 MiB +2.0% 15
Memory pluggy 45.9 MiB 47.6 MiB +3.7% 21
Memory werkzeug 135.8 MiB 138.7 MiB +2.1% 7
All benchmark scores
Mode Benchmark Base Head Change Runs
Memory requests 60.2 MiB 69.3 MiB +15.0% 4
Memory fastapi 146.1 MiB 148.0 MiB +1.3% 7
Memory httpx 81.0 MiB 81.9 MiB +1.1% 7
Memory h11 53.1 MiB 55.3 MiB +4.0% 21
Memory markupsafe 41.9 MiB 42.9 MiB +2.3% 21
Memory sniffio 47.6 MiB 47.9 MiB +0.6% 21
Memory itsdangerous 44.1 MiB 44.8 MiB +1.5% 21
Memory blinker 46.9 MiB 47.2 MiB +0.6% 21
Memory jinja 56.9 MiB 58.3 MiB +2.4% 21
Memory installer 48.6 MiB 48.9 MiB +0.6% 21
Memory tomlkit 49.1 MiB 50.0 MiB +2.0% 15
Memory outcome 48.6 MiB 48.8 MiB +0.5% 21
Memory pluggy 45.9 MiB 47.6 MiB +3.7% 21
Memory werkzeug 135.8 MiB 138.7 MiB +2.1% 7

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

✅ No diagnostic changes

Test result comparison
Project Previous New
requests ❌ 149 pass · 0 fail · 6 error · 12 skip ❌ 149 pass · 0 fail · 6 error · 12 skip
fastapi ❌ 1596 pass · 7 fail · 434 error · 1 skip ❌ 1596 pass · 7 fail · 434 error · 1 skip
httpx ❌ 1022 pass · 50 fail · 54 error · 1 skip ❌ 1022 pass · 50 fail · 54 error · 1 skip
h11 ✅ 78 pass · 0 fail · 0 error · 0 skip ✅ 78 pass · 0 fail · 0 error · 0 skip
markupsafe ❌ 0 pass · 0 fail · 26 error · 0 skip ❌ 0 pass · 0 fail · 26 error · 0 skip
sniffio ✅ 3 pass · 0 fail · 0 error · 1 skip ✅ 3 pass · 0 fail · 0 error · 1 skip
itsdangerous ✅ 10 pass · 0 fail · 0 error · 0 skip ✅ 10 pass · 0 fail · 0 error · 0 skip
blinker ✅ 25 pass · 0 fail · 0 error · 0 skip ✅ 25 pass · 0 fail · 0 error · 0 skip
jinja ❌ 135 pass · 0 fail · 16 error · 0 skip ❌ 135 pass · 0 fail · 16 error · 0 skip
installer ✅ 20 pass · 0 fail · 0 error · 0 skip ✅ 20 pass · 0 fail · 0 error · 0 skip
tomlkit ✅ 332 pass · 0 fail · 0 error · 0 skip ✅ 332 pass · 0 fail · 0 error · 0 skip
outcome ✅ 10 pass · 0 fail · 0 error · 0 skip ✅ 10 pass · 0 fail · 0 error · 0 skip
pluggy ❌ 124 pass · 1 fail · 27 error · 0 skip ❌ 124 pass · 1 fail · 27 error · 0 skip
werkzeug ❌ 728 pass · 3 fail · 0 error · 0 skip ❌ 728 pass · 3 fail · 0 error · 0 skip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test-running Related to running tests with karva

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rewrite assertions before test execution

1 participant