feat(markdown-normalizer): add intra-word emphasis fix for issue #97#100
Conversation
- New `enable_intra_word_emphasis_fix` Valve moves the opening emphasis marker to the start of a word when the wrapped content is pure punctuation, e.g. `series**,**` -> `**series,**` so added punctuation renders with the word (common in grammar/spelling diff outputs) - Supports **, __, ~~, ***, ___; code blocks and inline code are skipped - Bump version to 1.2.9; sync bilingual READMEs, docs mirror, and FEATURES_CN; add v1.2.9 release notes - Add tests under tests/ and plugins/.../tests/ covering the fix, code protection, and disabled-default behavior Closes #97
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
✅ Plugin Version Check / 插件版本检查版本更新检测通过！PR 包含版本变化和更新说明。 Version check passed! PR contains version changes and update description. 版本变化 / Version ChangesPlugin Updates
This comment was generated automatically. / 此评论由自动生成。 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
本拉取请求为 markdown_normalizer 过滤器插件引入了新功能“词内强调修复”（版本 v1.2.9），并新增了可选配置项 enable_intra_word_emphasis_fix，用于解决 LLM 在词边界内标记纯标点符号时（如 series**,**）Markdown 解析器无法正常渲染的问题。对此，评审人员提出了三项改进意见：首先，指出当前正则表达式中的 [^\\W_]+ 会排除下划线，导致 snake_case 变量名（如 my_var__,__）被错误拆分，建议将其替换为 \\w+；其次，指出 _fix_intra_word_emphasis 方法中的 while True 循环是多余的，因为该模式不会嵌套，单次 re.sub 即可，建议简化以节省资源；最后，建议在测试用例中增加对 snake_case 变量名的单元测试。
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| "intra_word_emphasis": re.compile( | ||
| r"(?P<word>[^\W_]+)" | ||
| r"(?P<marker>\*\*\*|___|\*\*|__|~~)" | ||
| r"(?P<content>[^\s\w]+?)" | ||
| r"(?P=marker)" | ||
| ), |
There was a problem hiding this comment.
问题分析：
在当前实现中，intra_word_emphasis 正则表达式使用了 [^\W_]+ 来匹配单词（word），这故意排除了下划线 _。
然而，在技术文本或代码中，下划线是 snake_case 变量名（如 my_var）的常见组成部分。
如果 LLM 输出形如 my_var__,__ 的内容（表示在变量名后添加了逗号），由于 _ 被排除，[^\W_]+ 只能匹配到 var，导致该正则匹配了 var__,__，最终将其错误地转换为 my___var,__。
这会把变量名拆开，导致渲染出来的格式极其混乱（my_ 正常，而 var, 被加粗），破坏了代码标识符的完整性。
解决方案：
建议将 [^\W_]+ 替换为 \w+。由于 content 部分已经通过 [^\s\w]+? 严格限制为纯标点符号（不含任何字母、数字或下划线），因此即使 word 匹配了包含下划线的完整变量名，也绝不会误匹配合法的 my_var__bold__（因为 bold 包含字母，无法匹配 content）。
使用 \w+ 后，my_var__,__ 将被完美修复为 __my_var,__，保持了变量名的完整性。
| "intra_word_emphasis": re.compile( | |
| r"(?P<word>[^\W_]+)" | |
| r"(?P<marker>\*\*\*|___|\*\*|__|~~)" | |
| r"(?P<content>[^\s\w]+?)" | |
| r"(?P=marker)" | |
| ), | |
| "intra_word_emphasis": re.compile( | |
| r"(?P<word>\w+)" | |
| r"(?P<marker>\*\*\*|___|\*\*|__|~~)" | |
| r"(?P<content>[^\s\w]+?)" | |
| r"(?P=marker)" | |
| ), |
| while True: | ||
| new_segment = self._PATTERNS["intra_word_emphasis"].sub( | ||
| replacer, inline_parts[k] | ||
| ) | ||
| if new_segment == inline_parts[k]: | ||
| break | ||
| inline_parts[k] = new_segment |
There was a problem hiding this comment.
问题分析：
在 _fix_intra_word_emphasis 方法中，使用了 while True 循环来反复执行 re.sub 直到字符串不再发生变化。
然而，由于 intra_word_emphasis 的匹配条件要求中间的 content 必须是纯标点符号（[^\s\w]+?），这意味着该模式绝不可能发生嵌套（因为标点符号内部不可能再包含单词和强调标记）。
因此，单次 re.sub 即可完全替换所有符合条件的匹配项，无需进行多次迭代。
使用 while True 不仅多余，还会导致每次处理时都至少额外执行一次无用的正则搜索（用于确认没有更多匹配），在高并发或长文本场景下会浪费 CPU 资源，甚至在极端情况下存在死循环的风险。
解决方案：
建议移除 while True 循环，直接进行单次替换。
inline_parts[k] = self._PATTERNS["intra_word_emphasis"].sub(
replacer, inline_parts[k]
)| # CJK word prefix | ||
| ("中文**,**", "**中文,**"), |
There was a problem hiding this comment.
建议：
为了验证我们对 snake_case 变量名（如 my_var__,__）修复逻辑的正确性，建议在此处增加对应的单元测试用例。
# CJK word prefix
("中文**,**", "**中文,**"),
# snake_case variable name with punctuation addition
("my_var__,__", "__my_var,__"),
|
Hi @dchurch315 👋 Thanks for the detailed feature request! I've opened PR #100 to address this exact use case. What's been implementedA new opt-in Valve Before → After
Code blocks and inline code are protected, and legitimate emphasis ( How to test
Known limitation (edge cases)During testing I noticed that when the punctuation marker sits inside an identifier-like token (e.g. Asking for your helpBefore I merge this, could you:
I'd rather get your real-world validation first than merge blindly. Happy to iterate based on your feedback! 🙏 — Fu-Jie |
|
It works but I notice that it only does so after all the content has been displayed, i.e. it doesn't change during the streaming. Not an issue for me but I wanted to mention it in case that wasn't expected behavior. I believe that you can handle stream events by switching outlet() to stream(), but I am a bit rusty. |
|
Hi @dchurch315, Glad to hear it works for your use case! 🎉 Regarding the streaming behavior you observed — that's actually expected and by design, not a bug. The Markdown Normalizer filter is intentionally implemented as a non-streaming post-processing filter via Why it's designed this wayThis plugin contains 9+ different fix passes (emphasis spacing, escape control, smart punctuation, HTML entity cleanup, intra-word emphasis, etc.), and almost all of them need to operate on the complete, final text to work correctly:
Why not switch to
|
1. Refine word pattern: `[^\W_]+` -> `[^\W_]+(?:_[^\W_]+)*`
- Snake_case identifiers (`my_var**,**`) now stay intact instead of
being split into `my_` + `var` (Gemini review, high priority).
- Trailing underscores are left for the marker alternation to consume,
so triple-underscore markers (`word___,___`) keep working correctly.
- Legitimate `my_var__bold__` is still protected (content excludes
word chars, so the pattern never matches).
2. Remove redundant `while True` loop in `_fix_intra_word_emphasis`.
- The pattern cannot nest (content is restricted to pure punctuation),
so a single `re.sub` pass is sufficient (Gemini review, medium).
3. Add snake_case test cases covering `my_var**,**`, `my_var__,__`,
`my_var~~,~~`, and `snake_case_var**.**`.
Test results: 110 passed, 3 failed (all 3 failures are pre-existing on
`main` and unrelated to this change).
|
Update for @dchurch315 and reviewers: I've pushed a follow-up commit ( What changed1. Snake_case identifiers now stay intact (Gemini review, high priority)The word pattern was refined from
Crucially, this is done without breaking triple-underscore markers — Legitimate 2. Removed redundant
|
2 participants
Summary
Adds a new opt-in
enable_intra_word_emphasis_fixValve to the Markdown Normalizer filter that fixes a long-standing rendering issue when LLMs mark up punctuation-only additions inside a word boundary (e.g. for grammar/spelling diff outputs). Closes #97.Background — Issue #97
The reporter uses Open WebUI for grammar and spelling checks and prefers the output to show deletions with
~~strikethrough~~and additions with**bold**. The problem: when an LLM marks up a punctuation-only addition inside a word boundary — e.g.series**,**to indicate the comma was added — most Markdown parsers refuse to render the inline emphasis because the opening delimiter is intra-word. The reporter asked for a toggle that moves the opening tag to the beginning of the word so the added punctuation renders together with the word.Changes
New Features
Intra-word Emphasis Repair (
enable_intra_word_emphasis_fix, defaultFalse): moves the opening emphasis marker to the start of a word when the wrapped content is pure punctuation:series**,****series,**word~~,~~~~word,~~word__,____word,__word***,******word,***Supports
**,__,~~,***,___. Only transforms cases where the wrapped content is pure punctuation (no whitespace, no word characters), so legitimate**bold**,word**bold**, math-likea**2**, and snake_case identifiers (my_var__bold__) are left untouched. Code blocks and inline code spans are also protected via the standardsplit("```")/split("")` partitioning.Safety / UX
enable_emphasis_spacing_fix).fix_intra_wordto all 12 supported languages.fix_key_mapso the status notification is translated correctly.Plugin Version
markdown_normalizerbumped tov1.2.9.Documentation
README.md,README_CN.md) updated with new "What's New in v1.2.9" section, feature description, and Valve table entry.docs/plugins/filters/markdown_normalizer.md,docs/plugins/filters/markdown_normalizer.zh.md, plus version badges inindex.md/index.zh.md.FEATURES_CN.mdextended with a new "11. 词内强调修复" section.v1.2.9.mdandv1.2.9_CN.md.Testing
plugins/filters/markdown_normalizer/tests/test_intra_word_emphasis.pycovering: bold/strikethrough/underscore/triple markers, multiple punctuation chars, multiple occurrences in a sentence, CJK word prefix, safe cases (legitimate emphasis, math, snake_case, HRs), code block + inline code protection, interaction withenable_emphasis_spacing_fix, and disabled-default behavior.tests/plugins/filters/markdown_normalizer/test_issue_97_regression.py(mirrors thetest_issue_57_regression.pypattern).tests/conftest.pywith anintra_word_only_normalizerfixture.test_emphasis_spacing.py::test_safeguard_and_correct_emphasis_unchanged[* italic *],test_reliability.py::test_code_block_escape_control) are unchanged onmainand unrelated to this PR.python3 scripts/check_version_consistency.pypasses.Testing Checklist
pytest plugins/filters/markdown_normalizer/tests/ tests/plugins/filters/markdown_normalizer/)Filter.outletintegration test verified变更摘要（中文）
为 Markdown Normalizer 过滤器新增可选启用的
enable_intra_word_emphasis_fix配置项，修复 LLM 在词边界内标记仅含标点的添加时（如语法/拼写差异输出场景）的渲染问题。Closes #97。新功能
enable_intra_word_emphasis_fix，默认False）：将开头的强调标记移动到词的开头，让添加的标点能与单词一起渲染。支持**、__、~~、***、___。仅对包含纯标点内容的情况生效，合法的**bold**/word**bold**/ 数学算式a**2**/ snake_case 标识符均不受影响；代码块与行内代码也受到保护。默认关闭
enable_emphasis_spacing_fix一致）。插件版本
markdown_normalizer升级到v1.2.9。文档
FEATURES_CN.md、v1.2.9.md/v1.2.9_CN.md发行说明均已同步。测试
tests/test_intra_word_emphasis.py与tests/plugins/filters/markdown_normalizer/test_issue_97_regression.py，覆盖：各类标记、纯标点内容、CJK、安全用例（合法强调/数学/snake_case/水平线）、代码块与行内代码保护、与enable_emphasis_spacing_fix的交互、默认关闭行为。check_version_consistency.py通过。