Subtle ABI mismatch in bundled SoftFloat library

[gottfriedleibniz]

The bundled SoftFloat64.lib defines bool as an int, following the definition in platform.h.in. This can lead to unexpected behavior in Hercules, as the upper bits of the int may remain uncleared as MSVC/clang-cl compiles bool as a one-byte type.

I first encountered this while compiling Hercules with clang-cl (versions 20.1.8 and 22.1.3) and running the test suite. For example:

Begin: "runtest -d $(DEV)\hyperion\tests\ -v quiet -64" ...

On:     Fri 04/17/2026  at 21:38:08 PM

Variable $platform           set to "Windows"
>>>>> line  2018: Received unexpected wait state:  0002000000000000 0000000000000BAD
Test "bfp-004-cvttolog":  0 OK compares.  One failure.
>>>>> line  2123: Received unexpected wait state:  0002000000000000 0000000000000BAD
Test "bfp-005-cvttolog64":  0 OK compares.  One failure.
>>>>> line  2291: Received unexpected wait state:  0002000000000000 0000000000000BAD
Test "bfp-006-cvttofix":  0 OK compares.  One failure.
>>>>> line  2456: Received unexpected wait state:  0002000000000000 0000000000000BAD
Test "bfp-007-cvttofix64":  0 OK compares.  One failure.
Did 305 tests.  4 failed; 301 OK

Investigating Test "bfp-004-cvttolog", I found that some failures were caused by f64_to_ui32 (in CLFDBR) setting the inexact bit in softfloat_exceptionFlags, even though it should have been suppressed. For reference, here is the (dis)assembly for f64_to_ui32 in SoftFloat64.lib:

.text:0000000000000034    mov     edi, r8d
...
.text:000000000000005D    mov     r9d, edi ;// 'bool' exact
.text:0000000000000060    movzx   r8d, sil ;// uint_fast8_t roundingMode
.text:0000000000000064    mov     rdx, rax ;// uint_fast64_t sig
.text:0000000000000067    mov     ecx, ebx ;// 'bool' sign
.text:0000000000000069    call    softfloat_roundToUI32

Jumping to softfloat_roundToUI32:

.text:0000000000000020    mov     ebx, r9d
// ...
.text:0000000000000109    test    ebx, ebx
.text:000000000000010B    jz      short loc_113

If bool were defined as a single byte, it is expected those two parameters would be setup with movzx ecx * and movzx r9d * for the softfloat_roundToUI32 call and the test would be 8-bit (or something isomorphic/functionally-equivalent).

In investigating why this bug has been dormant for so long I have provided the relevant snippets from:

Hercules (clang-cl 20.1.8):

.text:00000000000041A1    test    r14d, 400h ;// Multiple bytes packed in one register
.text:00000000000041A8    setz    r8b        ;// x86-64 does not automatically clear the upper bits of the register
.text:00000000000041AC    mov     rcx, rdi
.text:00000000000041AF    call    f64_to_ui32

Hercules (MSVC 19.51.36231):

.text:0000000000002F52    movzx   r8d, r12b ;// All higher bits of R8 are cleared!
.text:0000000000002F5B    shr     r8b, 2
// ...
.text:0000000000002F62    not     r8b
.text:0000000000002F65    and     r8b, 1
// ...
.text:0000000000002F6D    call    f64_to_ui32

This isn’t urgent; just a reminder to update the bool definition when dependencies are next bumped or recompiled.

[Fish-Git]

This can lead to unexpected behavior in Hercules, as the upper bits of the int may remain uncleared as MSVC/clang-cl compiles bool as a one-byte type.

Not in the MSVC case it doesn't.

In the MSVC case, only C++ treats bool as a single byte builtin type.

In C however, there is no bool builtin type defined. Thus we typedef it ourself from int.

The reason we typedef it from int instead of from BYTE (unsigned char) is because before our bool typedef existed, Hercules was always using poorly/unclearly named int variables to save the boolean result of a given comparison/condition, whereas declaring a variable with a more appropriately/clearly named variable of type bool instead, made the code less confusing and easier to understand and follow (and technically caused no change at all in actual functionality).

If clang is treating Hercules's typedef'ed bool variables as a single byte instead of as a 32-bit int, then that's a serious bug in clang that should be reported.

Am closing this issue as "WON'T FIX".

[gottfriedleibniz]

In the MSVC case, only C++ treats bool as a single byte builtin type.

In the MSVC C compiler (C99, C11, C17), the size of the native _Bool/bool type is 1 byte. For example, compiling MSVC latest w/ /TC /std:c11: https://godbolt.org/z/1hr1M1nr5

(Updated to C11 in the example to ensure strict compliance, given MSVC's historical baggage here).

[Fish-Git]

Re-Opening for further research...

[Fish-Git]

@gottfriedleibniz,

In the MSVC C compiler (C99, C11, C17), the size of the native _Bool/bool type is 1 byte.

Only for VS2013¹ or greater (yes?):

hyperion/hercwind.h

Lines 91 to 95 in ee86c4d

	#if _MSC_VER >= VS2013
	#define HAVE_INTTYPES_H
	#define HAVE_STDINT_H
	#define HAVE_STDBOOL_H
	#endif

For VS2008 however, we #define _Bool to int, and bool to _Bool, thus turning bool into an int as well:

hyperion/htypes.h

Lines 91 to 93 in ee86c4d

	#else
	#define _Bool int
	#define bool _Bool

Which is the same technique that all of the External Packages use when HAVE_STDBOOL_H is not defined.

This can lead to unexpected behavior in Hercules, as the upper bits of the int may remain uncleared as MSVC/clang-cl compiles bool as a one-byte type.

I believe however, as long as all of the External Packages (crypto, decNumber, SoftFloat and telnet) are all built using the same compiler version as Hercules itself is built with (which should be the normal case), then I believe everything should be fine. That is to say, there should not be any ABI mismatch, and everything should work as expected. (Yes?)

If Hercules is built with a completely different version of Visual Studio than what the External Packages are built with hwoever, then yes, there would be an ABI conflict. That I will grant you.

(Also, FYI: , I don't know Intel assembler (I apologize for that!), so the machine code you supplied in your original problem report, is gibberish to me. If you could point out where the actual bug is in your machine code (and explain what each instruction is doing, thus leading to the bug/problem), I might be better able to understand/appreciate your argument. Thanks!)

BOTTOM LINE:   I still don't believe there is any problem here (or if there is, I cannot see it!).

If you still believe there IS a problem, I would like to request 2 things:

1. Please help me to see and understand it, and...
2. Provide what you believe should be the appropriate fix.

Thanks!

Footnotes

1. Which I admit would be the normal case for those users who choose to build Hercules for themselves with a modern version of Visual Studio. ↩

[gottfriedleibniz]

Hi Fish, apologies for the delay (abrupt scheduling changes due to airline things)

I believe however, as long as all of the External Packages (crypto, decNumber, SoftFloat and telnet) are all built using the same compiler version as Hercules itself is built with (which should be the normal case), then I believe everything should be fine.

I think this depends on what you would consider the normal case to be. If you view hercules-helper as a companion repository, or as representative of how Hercules is typically built by others (i.e., not Release), on Linux that would certainly be the case. On Windows however, hercules-helper-windows does not build those packages and EXT_PKGS.msvc in this repository links directly against the libraries bundled.

(The following parts can be ignored depending on the response above).

If you could point out where the actual bug is in your machine code (and explain what each instruction is doing, thus leading to the bug/problem), I might be better able to understand/appreciate your argument. Thanks!)

It is not necessarily a bug but an artifact as to how clang and MSVC setup the calls into SoftFloat (R8 is the register holding the 'bool' argument). MSVC happens to clear the upper bits of R8 because it is less aggressive about packing multiple byte-sized C variables into a single register. As a result, it emits instructions that effectively zero out R8 except for the lowest byte (similar to casting to int). In contrast, clang-cl does not do this and leaves the upper bits of R8 undefined (it is significantly more aggressive in packing).

Provide what you believe should be the appropriate fix.

This is debatable given the above (and may require input from the hercules-helper-windows author). I only encountered this while building Hercules with clang-cl to run ASAN, so it is a very niche edge case that may not warrant immediate action (i.e., unless MSVC changes its behavior).

One example would be to wrap EXT_PKGS.msvc in a vsversion check similar to HERC_FLAGS.msvc.

[Fish-Git]

On Windows however, hercules-helper-windows does not build those packages and EXT_PKGS.msvc in this repository links directly against the libraries bundled.

(Oops!) You are of course correct! I forgot all about that! Those were all built using VS2008, where bool is always an int! (FRICK! Sorry about that!)

In contrast, clang-cl does not do this and leaves the upper bits of R8 undefined (it is significantly more aggressive in packing).

Well forgive me then gottfriedleibniz¹, but that sounds like a bug in clang to me! Yes?

Provide what you believe should be the appropriate fix.

This is debatable given the above (and may require input from the hercules-helper-windows author).

Yes, I believe you're probably right about that.²

One example would be to wrap EXT_PKGS.msvc in a vsversion check similar to HERC_FLAGS.msvc.

Great suggestion! I'll have to look into that! Thanks!

I'm glad I reopened this issue!

Footnotes

1. By the way, what is your first name?! Supplying it is purely optional of course (remaining anonymous is fine if that's what you really wish to do), but purely as a friendly courtesy, I personally prefer to call people by their first name when I converse with them rather than by a cold impersonal userid. ↩

2. His name is Bill Lewis (@wrljet) by the way. Really good guy. ↩

[gottfriedleibniz]

Well forgive me then gottfriedleibniz1, but that sounds like a bug in clang to me! Yes?

From VS2013+'s perspective since bool is defined as a single byte, the only ABI/contractual requirement is that the lowest byte is set; all higher bits are undefined. MSVC simply happens to clear the higher bits.

Great suggestion! I'll have to look into that! Thanks!

Another potential solution is eventually recompile all dependencies (e.g., bzip, zlib, etc.) with VS2013 or later and soft-deprecate older toolsets. Anyone wishing to build Hercules with an older toolset would need to recompile the dependencies themselves (well, mainly SoftFloat - all other libraries are ABI compatible).

In the grand scheme of things, this issue only arises in a very specific use-case. Since it is documented, it can be assigned a negative priority value.