Skip to content

Stop the Konsist gate passing without running, and land the Kotlin daemon measurement - #158

Merged
simtop merged 3 commits into
masterfrom
fix/konsist-gate-cannot-silently-pass
Aug 9, 2026
Merged

Stop the Konsist gate passing without running, and land the Kotlin daemon measurement#158
simtop merged 3 commits into
masterfrom
fix/konsist-gate-cannot-silently-pass

Conversation

@simtop

@simtop simtop commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Picks up the two commits that #157 did not take — it merged only its first commit, so the Kotlin daemon measurement and the Konsist heap fix never reached master — and then fixes the underlying pattern rather than another instance of it.

The Konsist gate is broken on master right now

A full ./gradlew :konsist:test --rerun-tasks on master dies with OutOfMemoryError in the test worker. Gradle forks test workers with a 512 MB heap and org.gradle.jvmargs does not reach them — that sizes the daemon, not the worker. These rules parse every .kt file in the repo into an in-memory Konsist model, and since #155 put the module on billionbeers.jvm.library it also carries jacoco instrumentation. maxHeapSize = "2g" fixes it.

It hid from both directions at once, which is why it survived review: a single-class run (--tests "*OneRuleTest*") fits in the default heap and passes, and an unchanged repo leaves the task UP-TO-DATE. Only a full, non-cached run reaches the ceiling — the run CI does.

The actual problem: this gate has now failed silently three times

  1. It never ran at allmake test targets testDebugUnitTest, which a pure-JVM module does not have.
  2. It went UP-TO-DATE while the code it guards changed, because the rules read the repo off the filesystem where Gradle cannot see it.
  3. It OOM'd on full runs while single-class runs passed (above).

Each got its own fix. Three instances of one shape is a pattern, so this adds a check for the shape: after the run, every *Test.kt rule file on disk must have produced a JUnit result class. A rule that exists but never executes now fails the build instead of reading as coverage — exactly what AGENTS.md warns about twice, and what nothing until now could detect.

It is deliberately self-maintaining. The expected set is derived from the files present rather than a hardcoded count, so adding a rule raises the bar automatically and the check cannot go stale. It also fails if it finds no rule files, since that would let both the check and the gate pass vacuously. A --tests filter skips it, because running one rule class is then the intent.

Verified with a mutation probe — a rule file with no @Test methods:

> These Konsist rules exist but did not run: GhostRuleTest.
  A rule that never executes looks like coverage and enforces nothing. Ran 13 of 14.

Removing the probe returns the build to green; a single-class --tests run stays green throughout.

Kotlin daemon heap, measured

kotlin.daemon.jvmargs was previously unset, so the JVM every module's Kotlin compilation runs through took an undocumented plugin default. Measured twice with the candidate order reversed, because one pass cannot separate a heap effect from the build warming up as the sweep proceeds:

heap 1st order reversed mean
1g 44.1s 40.7s 42.4s
1536m 37.9s 37.2s 37.6s
2g 32.6s 33.8s 33.2s
3g 30.1s 38.0s 34.1s

That control is load-bearing: 3g would have been the wrong answer from a single pass. It won the first ordering outright and came mid-pack in the second — an 8-second swing that tracks position in the list, not heap size. 1g is slowest even from the last slot where warming helps most, so it is genuinely too small. 2g is the only candidate fast in both orders.

Honest limitation, also written into gradle.properties: only wall time is trusted here. The Kotlin daemon exits the moment compilation ends and its GC log was not reliably captured, so its live-set and GC-overhead figures moved ~40% between identical runs and were not used to pick the value.

The measurement script's own guard was wrong twice

Worth recording, since it is the same failure shape as everything above — a check reporting success while verifying nothing:

  • pgrep -f PATTERN -l matches nothing on macOS (options must precede the pattern), so it reported "no Kotlin daemon found" while a daemon was running with exactly the requested heap.
  • Corrected, it still found nothing, because polling after the build is the wrong moment for a daemon that exits when compilation ends.

It now verifies the artifact instead of the process: if the daemon did not write the -Xlog:gc:file= it was given, the row is labelled NO-GC-LOG rather than reported as a confident zero. The script also detects monotonic-in-order wall times and tells you to re-run reversed — which is how the position effect above was caught.

Verification

make test, make konsist, make lint, spotlessCheck, plus the two mutation probes described above.

simtop added 3 commits August 9, 2026 15:55
… the first attempt

The Kotlin daemon is a second JVM that every module's Kotlin compilation runs
through, and it was previously unset. Sweeping it needed its own mode, since
reading the Gradle daemon's GC log while varying the Kotlin heap measures
nothing.

Measured twice with the candidate order reversed, because one pass cannot
separate a heap effect from the build warming up as the sweep proceeds:

  heap     1st order   reversed   mean
  1g          44.1s      40.7s    42.4s
  1536m       37.9s      37.2s    37.6s
  2g          32.6s      33.8s    33.2s   <- chosen
  3g          30.1s      38.0s    34.1s

That control is load-bearing. 3g looked like the clear winner in the first
order and mid-pack in the second - a position effect, not a heap one. 1g is
slowest even from the last slot where warming helps most, so it is really too
small. 2g is the only candidate fast in both orders.

Only wall time is trusted here, and the file says so: the daemon exits the
moment compilation ends and its GC log was not reliably captured, so its
live-set and GC figures moved ~40% between identical runs.

The guard was wrong twice, which is why the first sweep looked broken and the
second looked fine while reporting the same false alarm. `pgrep -f P -l`
matches nothing on macOS - options must precede the pattern. Corrected, it
still found nothing, because polling after the build is the wrong moment for a
daemon that has already exited. It now checks the artifact instead: if the
daemon did not write the GC log we asked it for, the row is labelled NO-GC-LOG
rather than reported as a confident zero.

The script also now flags a monotonic wall-time trend across candidates and
tells you to re-run reversed, which is how the position effect above was found.
A full `:konsist:test` died with "OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread Test worker". Gradle forks test workers with
a 512 MB heap and org.gradle.jvmargs does not reach them - that sizes the
daemon, not the worker. These rules parse every .kt file in the repo into an
in-memory Konsist model, and since the module joined billionbeers.jvm.library
it also carries jacoco instrumentation, which pushed it over.

It hid unusually well, in both directions at once. A single-class run
(`--tests "*OneRuleTest*"`) fits in the default heap and passes, which is how
the recent rule work was verified. An unchanged repo leaves the task
UP-TO-DATE, so `make konsist` returned green without executing anything. Only a
full, non-cached run reaches the ceiling - and that is the run CI does.

maxHeapSize = 2g. Verified by a --rerun-tasks run: 15 tests across 13 classes
execute and pass, where before the worker died before reporting any.
The architecture gate has now failed silently three separate ways. It never ran
at all, because `make test` targets testDebugUnitTest and a pure-JVM module has
no such task. It went UP-TO-DATE while the code it guards changed, because the
rules read the repo off the filesystem where Gradle cannot see it. And most
recently a full run died of OutOfMemory in the test worker while single-class
runs kept passing, so it looked green from every angle anyone checked.

Each of those got its own fix. This one addresses the shape rather than another
symptom: after the run, every `*Test.kt` rule file on disk must have produced a
JUnit result class. A rule that exists but never executes now fails the build
instead of reading as coverage - which is precisely what AGENTS.md warns about
twice and what nothing until now could detect.

It is deliberately self-maintaining. The expected set is derived from the files
present, not a hardcoded count, so adding a rule raises the bar automatically
and the check cannot go stale. It also fails if it finds no rule files at all,
since that would make both the check and the gate pass vacuously. A `--tests`
filter skips it, because running one rule class is then the intent.

Verified with a mutation probe: a rule file with no @test methods fails the
build with "These Konsist rules exist but did not run: GhostRuleTest. Ran 13 of
14", and removing it returns the build to green. A single-class --tests run
stays green throughout.
@simtop
simtop enabled auto-merge (squash) August 9, 2026 14:08
@simtop
simtop merged commit 8fc7ea3 into master Aug 9, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant