Skip to content

str encode/convert between utf8/16/etc., huge str split perf improvements, more functions#247

Merged
deviaze merged 4 commits into
mainfrom
str-encode-convert
Jul 10, 2026
Merged

str encode/convert between utf8/16/etc., huge str split perf improvements, more functions#247
deviaze merged 4 commits into
mainfrom
str-encode-convert

Conversation

@deviaze

@deviaze deviaze commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

82fc8bb — feat: str.encoding and str.convert

  • Merged std_str_internal.rs + std_str.luau into a unified src/std_str/ module (matching the mixed Rust+Luau pattern used elsewhere, e.g. std_io/prompt).
  • Added str.encoding(s) — detects Utf8/Utf8Bom/Utf16LE(Bom)/Utf16BE(Bom)/Binary from arbitrary bytes.
  • Added str.convert(s, to, from?) — converts between those encodings (built on encoding_rs for decode + hand-written UTF-16 encode, since encoding_rs deliberately doesn't support UTF-16 output).
  • Fixed issue trying to decode utf16 le json with BOM gives utf8 err #241 (UTF-16 JSON files failing to decode): json.readfile now detects encoding and normalizes to UTF-8 before parsing; json.decode (string input) still assumes UTF-8 for speed.

4931154 — perf: str.split/splitlines overhaul

  • Fixed issue str.split preallocates too much space on large strings #234 (crash on huge inputs): MAX_TABLE_SIZE was 1<<27, exactly 2x Luau's actual limit (1<<26, verified against Luau's own source).
  • Replaced per-item create_string+raw_push (which paid a mutex lock + stack guard per string) with batched create_sequence_from calls — ~2x win across the board.
  • Added a memchr fast path for single-byte separators (~7x faster matching than AhoCorasick for that case).
  • Rewrote str.splitlines as a dedicated Rust implementation (was str.split + a second Luau-side str.trimback pass) — net effect: ~8.9x faster (3.09s → 0.35s on a 175MB/5M-line benchmark), landing
    within ~1.3x of Python's C implementation.
  • Found and fixed perf gap caused by missing IntoLua impls for push_into_specified_stack fast path for BString/&BStr) — merged into mluau, giving ~1.5x on the non-UTF-8 fallback path.

40c52c2 — fix: str.splitlines correctness + str.width colors

  • Fixed str.splitlines silently dropping empty lines (inherited from str.split's "no empty splits" rule, which doesn't belong on a line-splitting function). Empty lines are now preserved, matching what most languages do.
  • Added include_terminator: boolean? param — controls whether a trailing line terminator with nothing after it produces a phantom trailing "".
  • Fixed str.width to strip ANSI color codes before measuring (previously every call site had to manually wrap with format.uncolor, duplicated across prompt.luau and the TUI example). inlined/translated format.uncolor to luau equivalent instead of cross module importing format into std/str luau code

1e34c3d — feat: more str functions

  • str.replace(s, old, new, max_replacements?, start_index?, end_index?) — literal (non-pattern) replace, SIMD substring search via memchr::memmem, with string.sub-style index bounds.
  • str.reverse(s) — grapheme-aware (Luau's builtin string.reverse corrupts multi-byte UTF-8).
  • str.rightpad(s, width, pad?) — companion to the existing str.leftpad.
  • str.splitfront/str.splitback(s, sep, max?) — bounded splitting from the front/back (max defaults to 1), equivalent to Python's str.split(sep, maxsplit)/str.rsplit(sep, maxsplit).
  • Fixed doc comments for str.chars/str.graphemes that incorrectly implied both work with str.slice/string.sub interchangeably — str.chars gives character positions (for str.slice), str.graphemes gives
    byte offsets (for string.sub); they aren't interchangeable.

Closes #241
Closes #234

@deviaze deviaze merged commit ec59620 into main Jul 10, 2026
10 of 12 checks passed
@deviaze deviaze deleted the str-encode-convert branch July 10, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant