Add CJK font fallback coverage#1552
Conversation
Load a generated CJK extension for the internal fallback font so dynamic text that misses the primary font can render common Chinese and Japanese notification/calendar content. Use Noto Sans CJK and a dedicated codepoint list rather than mutating the Chinese language pack list. Add tests for the CJK resource map, font glue, codepoint coverage, source font support, generated PBF glyph indexes, and Waf dependency tracking. Also track generated-font characterList files as Waf task inputs for TTF, OTF, and BDF resources so codepoint changes rebuild the generated PBF resource. Co-authored-by: GPT-5 Codex <codex@openai.com> Signed-off-by: Leask Wong <i@leaskh.com>
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR adds an extended CJK fallback font resource (with an explicit codepoint list) and wiring to ensure the font/resource pipeline and runtime fallback path can load it, plus a QEMU-only console command and tests to validate the setup.
Changes:
- Register a new
FONT_FALLBACK_INTERNAL_EXTENDEDfont resource backed by a CJK font + large codepoint list. - Update the font resource generator to track
characterListJSON files as build sources for non-PBF fonts. - Add tooling tests and a QEMU console command to validate CJK notification rendering and fallback behavior.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/tests/test_cjk_font_fallback.py | Adds end-to-end tests validating resource map entries, fallback wiring, codepoint coverage, and generated PBF indexing. |
| tools/resources/resource_map/resource_generator_font.py | Tracks characterList as a resolved source dependency for applicable font formats. |
| tools/font/fontgen.py | Fixes file-handle management when loading codepoint lists. |
| src/fw/wscript | Adjusts SRAM sizing behavior for “big platforms”, with special handling for QEMU. |
| src/fw/resource/system_resource.c | Wires the system fallback font loader to include an “extended” fallback resource id. |
| src/fw/console/prompt_commands.h | Exposes a QEMU-only test cjk notification prompt command. |
| src/fw/console/prompt_commands.c | Implements the QEMU-only CJK notification smoke test command. |
| resources/normal/base/resource_map.json | Registers the new extended fallback font resource. |
| resources/normal/base/lang/cjk_notification_codepoints.json | Adds the CJK notification codepoint list used to subset the extended fallback font. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| font_ext = os.path.splitext(d.file)[-1].lower() | ||
| if d.character_list and font_ext in (".ttf", ".otf", ".bdf"): | ||
| source_path = resource_source_path | ||
| if os.path.isabs(source_path): | ||
| source_path = os.path.relpath(source_path, bld.path.abspath()) | ||
| character_list_path = os.path.join(source_path, d.character_list) | ||
| if character_list_path.startswith(".."): | ||
| character_list_path = os.path.normpath(character_list_path) | ||
| else: | ||
| character_list_path = find_most_specific_filename( | ||
| bld, bld.env, bld.path, character_list_path | ||
| ) | ||
| d.character_list = character_list_path | ||
| dependency_path = character_list_path | ||
| if os.path.isabs(dependency_path): | ||
| dependency_path = os.path.relpath( | ||
| dependency_path, bld.path.abspath() | ||
| ) | ||
| d.sources.append(dependency_path) |
There was a problem hiding this comment.
Good catch. Fixed in 221e2dd: the characterList resolver now normalizes absolute paths first, checks the resolved path with commonpath against the resource source root, and keeps the fontgen open path separate from the waf dependency path. I also added tests for absolute in-tree paths, traversal outside the source root, and the resources-node build path that waf uses here.
| offset_table_format = "<" | ||
| offset_table_format += "L" if cp_bytes == 4 else "H" | ||
| offset_table_format += "H" if features & FEATURE_OFFSET_16 else "L" | ||
| offset_entry_size = struct.calcsize(offset_table_format) |
There was a problem hiding this comment.
Agreed. Fixed in 221e2dd by validating cp_bytes explicitly and raising an AssertionError unless the generated PBF uses 2- or 4-byte codepoints. That keeps this parser from silently interpreting unexpected values as uint16 codepoints.
| root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)) | ||
| sys.path.insert(0, root_dir) | ||
|
|
||
| import freetype # noqa: E402 |
There was a problem hiding this comment.
Agreed. Fixed in 221e2dd by wrapping the freetype/fontgen-dependent imports and skipping only the tests that need font generation when freetype is unavailable. I verified both paths: the project venv runs all 10 tests, while the system Python without freetype collects the file successfully and reports skips instead of failing import.
| "codepoints": [ | ||
| "164", | ||
| "167", | ||
| "168", |
There was a problem hiding this comment.
Agreed. Fixed in 221e2dd by converting the codepoint list to JSON numbers. The behavior is unchanged because fontgen already applies int(cp) while loading, but the source file is smaller and cheaper to parse.
Match QEMU big-platform SRAM regions to the production memory layout so the emulator boots with the same RAM split used by large boards. This keeps the QEMU CJK notification smoke path from depending on a stale memory shape. Signed-off-by: Leask Wong <i@leaskh.com>
Add a QEMU-only console command that injects a notification containing Chinese and Japanese text. The command gives reviewers a deterministic way to inspect system fallback font rendering in the emulator. Signed-off-by: Leask Wong <i@leaskh.com>
Refine the generated CJK fallback coverage and tests after review, keeping the codepoint set data-driven and making the font-resource tests easier to reason about. Signed-off-by: Leask Wong <i@leaskh.com>
Apply targetPlatforms filtering during resource map builds so generated resource definitions can be excluded from constrained boards. Limit the generated CJK fallback extension to emery and gabbro platforms. This keeps flint/asterix within its 1 MiB resource budget while preserving CJK coverage on larger resource targets. Signed-off-by: Leask Wong <i@leaskh.com>
Leask
force-pushed
the
codex/cjk-font-support
branch
from
221e2dd to
ab39948
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab3994839f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 8242, | ||
| 8243, | ||
| 8251, | ||
| 8451, |
There was a problem hiding this comment.
Adding U+2103 here does not make it render: prv_font_res_for_codepoint() only selects font_info->extension when !codepoint_is_latin(...) (src/fw/applib/graphics/text_resources.c:538), and codepoint_is_latin() treats U+2000–U+2BFF as Latin/symbols (src/fw/applib/fonts/codepoint.c:167). Since the base GOTHIC_14 fallback does not carry U+2103, CJK weather text such as 23℃ still falls through to the missing-glyph box despite being in this coverage file; either route missing base glyphs to the extension or keep these symbol-range glyphs in the base font.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch, this was a real routing bug. Fixed in ab8082c by retrying the loaded font extension when the preferred base font resource does not contain the requested codepoint. That keeps base glyphs taking priority while allowing symbol-range fallback glyphs such as U+2103 to render from the CJK extension instead of falling through to the missing-glyph box.
I also added a focused regression test for a symbol-range glyph that is absent from the base font but present in an extension resource, and verified it with:
./waf test -M test_text_resources --no_images --show_outputpython3 -m unittest tools.tests.test_cjk_font_fallback
Retry a loaded font extension when the preferred base font resource does not contain the requested codepoint. This lets symbol-range glyphs such as U+2103 render from the CJK fallback extension without adding them to the base font. Add a regression test using an existing symbol-range fixture glyph that is absent from GOTHIC_18 but present in an extension resource. Signed-off-by: Leask Wong <i@leaskh.com>
Summary
NotoSansCJKsc-DemiLight.otfand a seed-based notification/calendar coverage list.characterListinputs, so the generated PBF rebuilds when the CJK coverage seed changes.test cjk notificationconsole command for manual notification UI smoke testing.Design
The CJK resource is intentionally an extended fallback instead of replacing the existing built-in fallback. This keeps the normal ASCII/Latin path compact while allowing notification and calendar text to resolve missing Simplified Chinese, Traditional Chinese, and Japanese glyphs through the generated PBF.
The coverage list is seed-based rather than full pan-CJK. That keeps the resource within the normal pbpack budget while covering realistic notification/calendar text and the smoke-test strings used for Simplified Chinese, Traditional Chinese, Japanese Kanji, Hiragana, and Katakana.
The QEMU SRAM change is separate from text rendering, but it is needed for the validation path. qemu-pebble's M33 machines expose 480 KiB SRAM, while the previous linker layout assumed 511 KiB and could place memory past the mapped RAM.
Screenshot
Testing
.venv/bin/python -m unittest tools.tests.test_cjk_font_fallback.venv/bin/python ./waf build.venv/bin/python ./waf test -M test_text_resources --no_images --show_output.venv/bin/python ./waf test -M 'test_text_layout|test_word_iterator' --no_images --show_output./pbl qemu --qemu-decoration none, then./pbl consoleandtest cjk notification; verified Simplified Chinese, Traditional Chinese, Japanese Kanji, Hiragana, and Katakana render without tofu boxes, as shown above.Build budget after the change:
1584152 / 4194304bytes1045154 / 2097152bytes