Skip to content

Normalize all source files to NFC - #111

Merged
michakraus merged 1 commit into
mainfrom
chore/unicode-nfc-normalization
Jul 30, 2026
Merged

Normalize all source files to NFC#111
michakraus merged 1 commit into
mainfrom
chore/unicode-nfc-normalization

Conversation

@michakraus

@michakraus michakraus commented Jul 30, 2026

Copy link
Copy Markdown
Member

Follow-up to the review of #110, which turned up an encoding inconsistency that had been in the tree for a long time.

The problem

Accented identifiers in this repository had drifted into both of their legal spellings:

character composed (NFC) decomposed (NFD)
ū U+016B, 2 bytes u + U+0304 COMBINING MACRON, 3 bytes
U+1E21, 3 bytes g + U+0304, 3 bytes
U+1E57, 3 bytes p + U+0307, 3 bytes
ñ é U+00F1 / U+00E9, 2 bytes base + U+0303 / U+0301, 3 bytes

Julia's parser NFC-normalizes identifiers, so the two spellings are the same binding. Verified rather than assumed (with Ñ, whose two forms are the easiest to write out):

julia> nfd = "Ñ"; nfc = "Ñ"          # "4ecc83" vs "c391"
julia> eval(Meta.parse(nfd * "_probe = 7"))   # assign through the decomposed spelling
julia> eval(Meta.parse(nfc * "_probe"))       # read through the composed one
7
julia> filter(n -> occursin("_probe", string(n)), names(Main, all=true))
1-element Vector{Symbol}: :Ñ_probe            # one binding, not two

So a file mixing them compiles and runs identically, and no test of behaviour can distinguish them — which is exactly what let this sit. What they are not is the same text:

  • grep ū typed one way silently finds nothing while the identifier sits plainly on screen;
  • an exact-string edit fails for no visible reason;
  • a diff shows the line as modified while rendering byte-for-byte identically, so encoding churn is indistinguishable from a real edit.

Files drift between the forms without anyone touching a character, because macOS filesystem APIs hand back decomposed text and some editors recompose on save.

The change

Seven files, 31 lines, 44 combining marks folded away:

file characters
src/harmonic_oscillator.jl ū
src/lotka_volterra_2d_common.jl ū
src/lotka_volterra_2d_equations.jl ū
src/lotka_volterra_2d_symmetric.jl ñ
src/lotka_volterra_4d.jl
README.md, docs/make.jl é (in Hénon)

src/toda_lattice.jl was the eighth file until #110 landed: hand-writing the Toda vector fields rewrote its Ñ lines into NFC as a side effect, so this branch was rebased onto that and Ñ drops off the list. The rebase conflicted on exactly those seven lines and resolved in favour of #110's version, which is byte-identical to what normalization would have produced.

Every changed line is canonically equivalent to the one it replaces. Checked mechanically, not by eye: for each file, NFC(before) == after, and NFD(before) == NFD(after), so nothing but the normal form changed.

Only 15 bytes are removed rather than 44, because the composed forms of , and are themselves three-byte codepoints and cost exactly what their decomposed spellings did.

The guard

test/unicode_normalization_tests.jl keeps it this way. It asserts the normal form of each file rather than any particular character, so there is no list of accented identifiers to maintain, and it reports the offending paths rather than a count, so a failure says what to run normalize over.

Characters with no precomposed form pass untouched: is q + U+0307 and has no single codepoint, so it is already in normal form and no normalization pass can change it. obsolete/ is excluded, being kept for reference and not edited — its six files are the only decomposed text left in the tree.

Unicode is now a declared dependency in test/Project.toml, for the third time the same reason applies after Logging and LinearAlgebra: an undeclared stdlib resolves under julia --project=test but not inside the sandbox environment Pkg.test builds from that file, where using Unicode fails with "Package Unicode not found in current path". The pre-push hook caught this, which is the whole point of it.

Full Pkg.test() passes, as does the pre-push hook suite.

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings July 30, 2026 13:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 27.27273% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.22%. Comparing base (23f8c6d) to head (8a38e26).

Files with missing lines Patch % Lines
src/lotka_volterra_2d_common.jl 0.00% 9 Missing ⚠️
src/harmonic_oscillator.jl 0.00% 7 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #111   +/-   ##
=======================================
  Coverage   75.22%   75.22%           
=======================================
  Files          43       43           
  Lines        2442     2442           
=======================================
  Hits         1837     1837           
  Misses        605      605           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Accented identifiers in this repository had drifted into both of their legal
spellings. `u-macron` appeared as the single codepoint U+016B in some files and
as `u` followed by U+0304 COMBINING MACRON in others; likewise `g-macron`,
`p-dot`, `n-tilde`, and the `e-acute` of Henon in README.md and docs/make.jl.

Julia's parser NFC-normalizes identifiers, so the two spellings are the same
binding: a file mixing them compiles and runs identically, and no test of
behaviour can distinguish them. That is exactly what makes this worth fixing
deliberately rather than never. They are not the same *text*: grep for the
composed form silently finds nothing while the identifier sits plainly on
screen, and an exact-string edit fails for no visible reason. Files drift
between the forms without anyone touching a character, because macOS filesystem
APIs hand back decomposed text and some editors recompose on save.

Seven files, 31 lines, 44 combining marks folded away -- every line canonically
equivalent to the one it replaces, verified by checking that NFD-folding both
sides gives identical text:

  src/harmonic_oscillator.jl              u-macron g-macron p-dot
  src/lotka_volterra_2d_common.jl         u-macron g-macron p-dot
  src/lotka_volterra_2d_equations.jl      u-macron g-macron
  src/lotka_volterra_2d_symmetric.jl      n-tilde
  src/lotka_volterra_4d.jl                p-dot
  README.md, docs/make.jl                 e-acute

src/toda_lattice.jl was the eighth until #110 rewrote its N-tilde lines into NFC
as a side effect of hand-writing the Toda vector fields, which is why N-tilde is
absent above.

Only 15 bytes are removed rather than 44, because the composed forms of p-dot,
g-macron and x-dot are themselves three-byte codepoints and so cost exactly what
their decomposed spellings did.

test/unicode_normalization_tests.jl keeps it this way. It asserts the normal form
of each file rather than any particular character, so there is no list of
accented identifiers to maintain, and it names the offending files rather than
counting them, so a failure says what to run `normalize` over. Characters with no
precomposed form pass untouched: `q-dot` is `q` plus U+0307 and has no single
codepoint, so it is already in normal form. `obsolete/` is excluded, being kept
for reference and not edited; its six files are the only decomposed text left in
the tree.

`Unicode` joins `test/Project.toml`, for the third time the same reason applies
after `Logging` and `LinearAlgebra`: an undeclared stdlib resolves under
`julia --project=test` but not inside the sandbox environment `Pkg.test` builds
from that file, where `using Unicode` fails with "Package Unicode not found in
current path". The pre-push hook caught it, which is the whole point of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@michakraus
michakraus force-pushed the chore/unicode-nfc-normalization branch from 9fac4ff to 8a38e26 Compare July 30, 2026 13:52
@michakraus
michakraus merged commit 465741a into main Jul 30, 2026
11 of 15 checks passed
@michakraus
michakraus deleted the chore/unicode-nfc-normalization branch July 30, 2026 15:23
michakraus added a commit that referenced this pull request Jul 30, 2026
v0.8.0 was tagged at 52c123e, but three PRs landed after that tag while their
entries were written into the `## [0.8.0]` section: #110 (hand-written Toda
vector fields), #111 (NFC normalization, which had its own `## [Unreleased]`
section) and #112 (the ensemble size assertion). So 0.8.0's section described
nine bullets that v0.8.0 does not contain.

Split by which bullets existed at the v0.8.0 tag, the same way 52c123e split
0.7.4 out of 0.8.0:

  ## [0.8.1] — 2026-07-30   Added (3), Changed (3), Documentation (1), Tests (2),
                            Repository hygiene (1), Known follow-ups
  ## [0.8.0] — 2026-07-30   unchanged but for the nine bullets that moved out

Four cross-references follow from the move, each having pointed at a neighbour
that stayed behind in 0.8.0:

  * "the last of the three conversions" becomes "completing the three
    conversions 0.8.0 began with OuterSolarSystem and LinearWave";
  * "gained the same four-argument methods" becomes "gained the four-argument
    methods LinearWave gained in 0.8.0";
  * the lode_wiring entry's "for the same reason" now names the reason, rather
    than pointing at a LinearWave bullet in the previous section;
  * "the same trap the linear wave hit above" becomes "in 0.8.0".

0.8.0's `Changed` preamble says "unlike the two signature repairs below", which
#110 had made wrong by adding a third; moving the Toda entry out makes the count
correct again.

`Known follow-ups` moves with the release, per 52c123e's convention of listing
the open ones once, under the newest section. Both remaining items are still
open.

The compare-link block gains `[0.8.1]`, `[0.8.0]` and `[0.7.4]`, which were
never added, and `[Unreleased]` is repointed from v0.7.3 to v0.8.1.

Version 0.8.1 rather than 0.9.0: no exported signature changed, the removed
`const Omega` was never exported, and the tightened size assertion rejects input
that previously produced wrong answers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants