fix(linter): exclude string-literal contents from paren/quote balance checks#50
Conversation
… checks
The linter counted parentheses and quotes inside string literals,
producing false positive "unbalanced" warnings for code like छपाउ("(क").
Use _tokenize_preserving_strings from the transpiler to extract only
the code portions of each line before counting. Also add single-quote
balance checking (previously only double quotes were checked).
Regression tests cover: parens/quotes inside strings (no error),
genuinely unbalanced lines (still error), single-quote balance,
apostrophe in double-quoted strings.
Closes alphacrack#30
alphacrack
left a comment
There was a problem hiding this comment.
Really solid contribution — the fix reuses _tokenize_preserving_strings exactly as #30 suggested, adds the missing single-quote balance check, registers against the existing security marker correctly (the project runs --strict-markers, and security is declared in pyproject, so you're safe), and includes genuine negative controls. Thank you!
One test needs a fix before merge — it silently doesn't test what its name says:
def test_quote_inside_string_not_flagged():
code = 'छपाउ("it''s ok")\n'Inside a single-quoted Python literal, '' closes and reopens the string — adjacent-literal concatenation. The value is actually छपाउ("its ok") (verified: repr(code) shows no apostrophe), so the apostrophe-inside-double-quotes case is never exercised. The underlying behavior is correct (I checked: छपाउ("it's ok") lints clean with your change), the test just needs to construct the string it claims to:
code = 'छपाउ("it\'s ok")\n' # escaped apostropheStyle suggestion while you're in there (non-blocking): the rest of test_linter.py uses literal Devanagari rather than \uXXXX escapes — literals would make these tests much easier to read for the Maithili-speaking contributors this project hopes to attract.
Happy to approve as soon as the one-line test fix lands. CI run is approved on my end.
|
Thanks for catching that. You were right—the adjacent string literals removed the apostrophe, so the test wasn't exercising the intended case. I've corrected the test, added an assertion that verifies the apostrophe is present in the constructed input, reran the focused linter/security tests, and pushed the update. |
Problem
The linter in
maithili_dsl/transpiler/linter.pyflagged false warnings for unbalanced parentheses and quotes when those characters appeared inside string literals. For example, valid code likeछपाउ("(क")producedपंक्ति 1: गोल ब्रैकेट असंतुलित अछि.Additionally, only double-quote balance was checked; single quotes were never validated at all.
Root cause
lint_maithili_codeusedstripped.count("(")andstripped.count('"')on the entire line including string literal contents. Characters inside"..."and'...'strings should not contribute to the balance count.Change
_tokenize_preserving_stringsfrommaithili_dsl.transpiler.transpile— the same string-aware tokenizer already used by the transpiler.') balance checking alongside the existing double-quote check.Files changed:
maithili_dsl/transpiler/linter.py(+12/-4, functional changes only)tests/test_linter.py(+6 regression tests)tests/test_security.py(+2 security regression tests)CHANGELOG.md(Unreleased entry)Tests
Before fix:
lint_maithili_code('छपाउ("(क")\n')→['पंक्ति 1: गोल ब्रैकेट असंतुलित अछि'](false positive)After fix: same input returns
[](no errors). Genuinely unbalanced parens and quotes are still caught.New tests:
test_parens_inside_double_quoted_string_not_flaggedtest_parens_inside_single_quoted_string_not_flaggedtest_quote_inside_string_not_flaggedtest_unbalanced_single_quote_flagged(new capability)test_genuinely_unbalanced_parens_still_flaggedtest_genuinely_unbalanced_quotes_still_flaggedtest_linter_tokenizer_does_not_expand_trust_boundary(security)test_actual_dangerous_code_outside_strings_still_blocked(security)Test results: 152 passed. 5 pre-existing Windows smoke-test failures (cp1252 encoding of Devanagari subprocess output) confirmed identical on unmodified
development.Security Considerations
This change only reuses the existing string-aware tokenizer for lint balance checks. It does not alter transpilation, import validation, or sandbox execution. Security regression tests in
tests/test_security.pyconfirm that:_validate_importsgate.The trust boundary is unchanged.
Risk
Minimal. The tokenizer is the same function used by the transpiler's keyword-replacement path, which has existing test coverage for string literal preservation. No new dependencies, no API change.
Closes #30