Pony's standard library has no notion of grapheme clusters. Strings expose bytes and codepoints, but a user-perceived character — a grapheme cluster, per Unicode UAX #29 — can span multiple codepoints (combining marks, emoji sequences, and the like). Code that needs to reason about "characters" the way a user sees them has nothing to build on.
This surfaced in ponylang/ponyc#3125, where the format package computes display width from byte length and gets it wrong for multi-byte characters. Switching to codepoints() handles the simplest cases but still breaks on grapheme clusters and on characters whose display width isn't one. Doing it properly means implementing grapheme cluster boundary detection, which requires handling and parsing the Unicode data files — similar to the approach other languages such as Elixir take.
We'd love to see someone take up designing such changes and writing the RFC.
Origin
This came out of ponylang/ponyc#3125.
Pony's standard library has no notion of grapheme clusters. Strings expose bytes and codepoints, but a user-perceived character — a grapheme cluster, per Unicode UAX #29 — can span multiple codepoints (combining marks, emoji sequences, and the like). Code that needs to reason about "characters" the way a user sees them has nothing to build on.
This surfaced in ponylang/ponyc#3125, where the
formatpackage computes display width from byte length and gets it wrong for multi-byte characters. Switching tocodepoints()handles the simplest cases but still breaks on grapheme clusters and on characters whose display width isn't one. Doing it properly means implementing grapheme cluster boundary detection, which requires handling and parsing the Unicode data files — similar to the approach other languages such as Elixir take.We'd love to see someone take up designing such changes and writing the RFC.
Origin
This came out of ponylang/ponyc#3125.