UTF-8 toolchain

[CyanoHao]

General discussion for UTF-8 toolchain.

UTF-8 support in MinGW Lite:

• UTF-8 manifest (general introduction, the manifest in that page is INCORRECT; correct manifest): injected to all executables for branch 15, 14, 13. On Windows 10 1903+, UTF-8 manifest make Windows API to interpret char * parameters as UTF-8 encoded strings. Thus the toolchain gets Unicode support on Windows 10 1903+.
• UTF-8 thunk: applied to all executables for branch 16+. Since char * strings come from and go to API boundary, by replacing -A APIs (more accurate, __imp_ symbols in import library) with thunks that do UTF-8 <-> UTF-16 conversion and call -W APIs (example), the toolchain gets full Unicode support on NT family, and best effort on 9x.

Beyond MinGW Lite:

• UTF-8 thunk can be applied to not only the toolchain executables, but also target import libraries and shared libraries delivered to user.
• Windows provide too many APIs. It is not practical to provide thunks for all of them. But what if we limit thunks to CRT and STL? That is the “u8crt” profiles in MinGW Lite’s build system, planned for MinGW ∞ project.

[CyanoHao]

Additional information.

Related discussion: skeeto/w64devkit#315.

Timeline

2022

1. royqh1979 noticed that ld fails to generate executable with non-ASCII filename when trying to bump bundled toolchain in Red Panda C++ to GCC 12.x (MinGW Builds).

2023

1. GCC 13 came with UTF-8 manifest.
2. skeeto noticed that GCC 13.1 does not work on Windows XP (skeeto/w64devkit#58) and shortly fixed it by disabling UTF-8 manifest.
3. I (CyanoHao) invested the ld failure with non-ASCII filename and found 2 bugs (niXman/mingw-builds#649).

2024

1. anbangli asked for GCC NLS (niXman/mingw-builds#666).
2. I did some investigation about GCC NLS in MinGW Builds. The build process costs hours, so MinGW Lite was created for easy experiments (commit 1d9bf2d), based on w64devkit.
3. GCC NLS is implemented in MinGW Lite (started in a separate git branch, merged to git master later as commit 96131d8).
4. Windows 2000 support is added to MinGW Lite, by patching GCC and winpthreads and using early gdb version (commit 64fc15b).
5. NLS exposed GCC’s mismatching console code page. After this got fixed (commit 10aa9bd), GCC 14 removed bundled libintl (gettext), and that patch broke standalone libintl, so it got fixed again (commit e4e0c47) ...
6. Current version of gdb was ported to Windows 2000, by adding thunks for missing functions (commit 64e3d67, inspired by YY-Thunks).
7. Windows NT 4.0 and partial Windows 9x support were added, by adding more patches and thunks (commit e5691a8).

2025

1. Thunks were converted to a patch set for mingw-w64 (commit d9b8c70). Legacy OS support was temporarily disabled and later restored.
2. ... And again libintl (commit 30be624) ...
3. anbangli reported g++ and ld have different encoding in their error messages. I realized that it is impossible for IDE to decode error message unless all tools had consistent encoding. UTF-8 manifest was injected to all executables (commit e69c5ec).
4. ... And again libintl. I got tired of patching tools. Instead, I wrote an alternative libintl, which simply infer language code from GetUserPreferredUILanguages (commit f555638).
5. Peter0x44 fixed GCC’s mismatching console code page (skeeto/w64devkit#279).
6. royqh1979 noticed compile error on Windows 7 when the toolchain was installed to non-ASCII path (commit 652fba3).
7. royqh1979 noticed mojibake on Windows 7, for example “版本” (GBK: \xb0\xe6\xb1\xbe) → “苞纡柄菌” (GBK: \xb0\xfa\xe6\xfa\xb1\xfa\xbe\xfa). I dug into a UCRT bug in write.cpp (was source_it < buffer_end) and wrote a patch (drop commit 71a3c0b). The patch unfortunately (or fortunately) broke all 64-bit NT 5.2 profiles. The 2 sequential problems drove me to check Microsoft’s document for UTF-8 manifest, and I found the mistake (commit c55534d, skeeto/w64devkit#315).

2026

1. A Red Panda C++ user mentioned #include "non-ASCII name", and I encountered error in mcfgthread #include __FILE__. I realized full Unicode support is a better way to fix those UTF-8 source code <-> ACP API boundary bugs (commit 6f259b1).