Normalize all source files to NFC - #111
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #111 +/- ##
=======================================
Coverage 75.22% 75.22%
=======================================
Files 43 43
Lines 2442 2442
=======================================
Hits 1837 1837
Misses 605 605 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Accented identifiers in this repository had drifted into both of their legal spellings. `u-macron` appeared as the single codepoint U+016B in some files and as `u` followed by U+0304 COMBINING MACRON in others; likewise `g-macron`, `p-dot`, `n-tilde`, and the `e-acute` of Henon in README.md and docs/make.jl. Julia's parser NFC-normalizes identifiers, so the two spellings are the same binding: a file mixing them compiles and runs identically, and no test of behaviour can distinguish them. That is exactly what makes this worth fixing deliberately rather than never. They are not the same *text*: grep for the composed form silently finds nothing while the identifier sits plainly on screen, and an exact-string edit fails for no visible reason. Files drift between the forms without anyone touching a character, because macOS filesystem APIs hand back decomposed text and some editors recompose on save. Seven files, 31 lines, 44 combining marks folded away -- every line canonically equivalent to the one it replaces, verified by checking that NFD-folding both sides gives identical text: src/harmonic_oscillator.jl u-macron g-macron p-dot src/lotka_volterra_2d_common.jl u-macron g-macron p-dot src/lotka_volterra_2d_equations.jl u-macron g-macron src/lotka_volterra_2d_symmetric.jl n-tilde src/lotka_volterra_4d.jl p-dot README.md, docs/make.jl e-acute src/toda_lattice.jl was the eighth until #110 rewrote its N-tilde lines into NFC as a side effect of hand-writing the Toda vector fields, which is why N-tilde is absent above. Only 15 bytes are removed rather than 44, because the composed forms of p-dot, g-macron and x-dot are themselves three-byte codepoints and so cost exactly what their decomposed spellings did. test/unicode_normalization_tests.jl keeps it this way. It asserts the normal form of each file rather than any particular character, so there is no list of accented identifiers to maintain, and it names the offending files rather than counting them, so a failure says what to run `normalize` over. Characters with no precomposed form pass untouched: `q-dot` is `q` plus U+0307 and has no single codepoint, so it is already in normal form. `obsolete/` is excluded, being kept for reference and not edited; its six files are the only decomposed text left in the tree. `Unicode` joins `test/Project.toml`, for the third time the same reason applies after `Logging` and `LinearAlgebra`: an undeclared stdlib resolves under `julia --project=test` but not inside the sandbox environment `Pkg.test` builds from that file, where `using Unicode` fails with "Package Unicode not found in current path". The pre-push hook caught it, which is the whole point of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
michakraus
force-pushed
the
chore/unicode-nfc-normalization
branch
from
July 30, 2026 13:52
9fac4ff to
8a38e26
Compare
michakraus
added a commit
that referenced
this pull request
Jul 30, 2026
v0.8.0 was tagged at 52c123e, but three PRs landed after that tag while their entries were written into the `## [0.8.0]` section: #110 (hand-written Toda vector fields), #111 (NFC normalization, which had its own `## [Unreleased]` section) and #112 (the ensemble size assertion). So 0.8.0's section described nine bullets that v0.8.0 does not contain. Split by which bullets existed at the v0.8.0 tag, the same way 52c123e split 0.7.4 out of 0.8.0: ## [0.8.1] — 2026-07-30 Added (3), Changed (3), Documentation (1), Tests (2), Repository hygiene (1), Known follow-ups ## [0.8.0] — 2026-07-30 unchanged but for the nine bullets that moved out Four cross-references follow from the move, each having pointed at a neighbour that stayed behind in 0.8.0: * "the last of the three conversions" becomes "completing the three conversions 0.8.0 began with OuterSolarSystem and LinearWave"; * "gained the same four-argument methods" becomes "gained the four-argument methods LinearWave gained in 0.8.0"; * the lode_wiring entry's "for the same reason" now names the reason, rather than pointing at a LinearWave bullet in the previous section; * "the same trap the linear wave hit above" becomes "in 0.8.0". 0.8.0's `Changed` preamble says "unlike the two signature repairs below", which #110 had made wrong by adding a third; moving the Toda entry out makes the count correct again. `Known follow-ups` moves with the release, per 52c123e's convention of listing the open ones once, under the newest section. Both remaining items are still open. The compare-link block gains `[0.8.1]`, `[0.8.0]` and `[0.7.4]`, which were never added, and `[Unreleased]` is repointed from v0.7.3 to v0.8.1. Version 0.8.1 rather than 0.9.0: no exported signature changed, the removed `const Omega` was never exported, and the tightened size assertion rejects input that previously produced wrong answers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the review of #110, which turned up an encoding inconsistency that had been in the tree for a long time.
The problem
Accented identifiers in this repository had drifted into both of their legal spellings:
ūu+ U+0304 COMBINING MACRON, 3 bytesḡg+ U+0304, 3 bytesṗp+ U+0307, 3 bytesñéJulia's parser NFC-normalizes identifiers, so the two spellings are the same binding. Verified rather than assumed (with
Ñ, whose two forms are the easiest to write out):So a file mixing them compiles and runs identically, and no test of behaviour can distinguish them — which is exactly what let this sit. What they are not is the same text:
grep ūtyped one way silently finds nothing while the identifier sits plainly on screen;Files drift between the forms without anyone touching a character, because macOS filesystem APIs hand back decomposed text and some editors recompose on save.
The change
Seven files, 31 lines, 44 combining marks folded away:
src/harmonic_oscillator.jlūḡṗsrc/lotka_volterra_2d_common.jlūḡṗsrc/lotka_volterra_2d_equations.jlūḡsrc/lotka_volterra_2d_symmetric.jlñsrc/lotka_volterra_4d.jlṗREADME.md,docs/make.jlé(in Hénon)src/toda_lattice.jlwas the eighth file until #110 landed: hand-writing the Toda vector fields rewrote itsÑlines into NFC as a side effect, so this branch was rebased onto that andÑdrops off the list. The rebase conflicted on exactly those seven lines and resolved in favour of #110's version, which is byte-identical to what normalization would have produced.Every changed line is canonically equivalent to the one it replaces. Checked mechanically, not by eye: for each file,
NFC(before) == after, andNFD(before) == NFD(after), so nothing but the normal form changed.Only 15 bytes are removed rather than 44, because the composed forms of
ṗ,ḡandẋare themselves three-byte codepoints and cost exactly what their decomposed spellings did.The guard
test/unicode_normalization_tests.jlkeeps it this way. It asserts the normal form of each file rather than any particular character, so there is no list of accented identifiers to maintain, and it reports the offending paths rather than a count, so a failure says what to runnormalizeover.Characters with no precomposed form pass untouched:
q̇isq+ U+0307 and has no single codepoint, so it is already in normal form and no normalization pass can change it.obsolete/is excluded, being kept for reference and not edited — its six files are the only decomposed text left in the tree.Unicodeis now a declared dependency intest/Project.toml, for the third time the same reason applies afterLoggingandLinearAlgebra: an undeclared stdlib resolves underjulia --project=testbut not inside the sandbox environmentPkg.testbuilds from that file, whereusing Unicodefails with "Package Unicode not found in current path". The pre-push hook caught this, which is the whole point of it.Full
Pkg.test()passes, as does the pre-push hook suite.🤖 Generated with Claude Code