Switch buffer from bytes to characters for proper multibyte handling#100
Open
tk0miya wants to merge 2 commits into
Open
Switch buffer from bytes to characters for proper multibyte handling#100tk0miya wants to merge 2 commits into
tk0miya wants to merge 2 commits into
Conversation
The RubyRenderer was using byte-based positions for buffer operations, but RuboCop uses character-based positions. This caused linter offenses to be reported at incorrect line/column positions when the source file contained multibyte characters before the offense location. Changes: - Add byte_to_char_pos method to Source class for converting Herb's byte positions to character positions - Change RubyRenderer buffer from Array[Integer] (bytes) to Array[String] (characters) - Update all buffer operations to use character positions instead of byte positions - Add string constants (CHAR_*) to Characters module for character-based buffer operations - Update Converter.generate_hybrid_code to work with character positions - Update tests to check character length instead of byte length
- Add slice() and byte_range_to_char_range() methods to Source - Change Tag to store char_from/char_to instead of byte-based range - Update NodeLocationCollector to create Tags with character positions - Update Converter to use slice() for character-based extraction - Update RuboCopASTTransformer to use character-based tag lookup This follows up on the multibyte position fix by ensuring all position data is stored in character units rather than mixing byte and character positions, making the codebase easier to work with.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR refactors the code rendering system to work with character positions instead of byte positions. This fixes incorrect handling of multibyte characters (e.g., emoji, CJK characters) in ERB templates.
Key Changes
Buffer representation: Changed from
Array[Integer](bytes) toArray[String](characters)buffer.joininstead ofpack("C*")Character constants: Added string equivalents for all byte constants in
CharactersmoduleCHAR_LF,CHAR_CR,CHAR_SPACE,CHAR_HASH,CHAR_SEMICOLON,CHAR_EQUALS,CHAR_UNDERSCORE,CHAR_LEFT_BRACE,CHAR_RIGHT_BRACEPosition conversion: Implemented
byte_to_char_posmethod inSourceclassParseResultandRubyRendererConverter refactoring: Updated
generate_hybrid_codeto work with character arraysbyte_to_char_posto convert positions before buffer manipulationRubyRenderer updates: All position calculations now use character positions
render_code_node,render_output_marker,render_open_tag_node,render_close_tag_node,render_text_node,render_erb_comment_node,render_html_comment_node,render_tag_markerall updatedbleach_codenow returnsArray[String]and uses character-based logicTest updates: Fixed test expectations to use character counts instead of byte counts
Implementation Details
The core insight is that RuboCop operates on character positions, while Herb's parser provides byte positions. By converting the buffer to character-based operations, we eliminate the mismatch that caused incorrect positioning with multibyte characters. The
byte_to_char_posconversion is applied at the boundary where Herb positions are used in RuboCop's buffer.