Point the efficiency tests at the engine's actual name - #804
Open
ZacharyZcR wants to merge 1 commit into
Open
Conversation
tests/test_inefficiency.py looked for `c/glm.exe`. The engine was renamed
glm -> colibri, and .exe only exists on Windows, so that path resolves on no
platform at all: _engine_present() has been returning False everywhere and
all 8 tests in the file have been skipping instead of running.
$ python3 -m unittest discover -s tests -p 'test_inefficiency.py' -v
... skipped 'glm.exe not built (run: make glm.exe)' x8
They are the throughput floor, the disk-wait share ceiling, the PROFILE
phase assertions and the CPU-vs-CUDA teacher-forcing agreement -- the
regression tests for the engine being fast, which is the property this
project exists for. A green `make test-python` has been reporting on none
of it.
tools/efficiency.py had the same default (`../glm.exe`), so anything else
driving it landed on the same missing file.
Both now resolve colibri/colibri.exe, preferring whichever exists. With the
engine built, the skip reason becomes the honest one:
... skipped 'glm_tiny fixture absent (... run tools/make_glm_oracle.py)'
which is a real prerequisite -- the fixture is gitignored and needs torch to
generate -- rather than a name nobody updated.
CONTRIBUTING's oracle command had the same stale binary and never said where
glm_tiny comes from, so a new contributor following it hits two dead ends in
one line.
This was referenced Aug 3, 2026
ZacharyZcR
marked this pull request as ready for review
August 3, 2026 21:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft.
The efficiency tests have never run
tests/test_inefficiency.pylooks forc/glm.exe. The engine was renamedglm→colibri, and.exeonly exists on Windows — so that path resolves on no platform at all:_engine_present()has been returning False everywhere, so all 8 tests skip instead of running. A greenmake test-pythonreports on none of them.What they assert is not incidental — it is the property this project exists for:
TINY_TOK_S_FLOOR)PROFILEphase present and non-negativetools/efficiency.pyhad the same default (../glm.exe), so anything else driving it landed on the same missing file.After
Both resolve
colibri/colibri.exe, preferring whichever exists. With the engine built, the skip reason becomes the honest one:... skipped 'glm_tiny fixture absent (gitignored; ... run tools/make_glm_oracle.py)'That is a real prerequisite — the fixture is generated and needs torch — rather than a name nobody updated. The tests are still skipped in CI by design (
check.ymlis "no model downloads", #140), but they now run for anyone who has the fixture, which was the intent.CONTRIBUTING had the same stale binary
Two dead ends in one line for a new contributor:
./glmdoes not exist (themake glmalias survives, the binary iscolibri), andglm_tinyis gitignored with no hint about where it comes from. Fixed, and it now says the fixture is generated bytools/make_glm_oracle.py.docs/windows.md:124had already been updated to./colibri.exe— this one was simply missed.Scope
Names only. No test logic, no thresholds, no engine code. The tests were written to run and could not; this lets them.