buildsystem: enable LTO by default

[benpicco]

Contribution description

Let's see where we are at this now.

Testing procedure

Issues/PRs references

Replaces #14795.

Declaration of AI-Tools / LLMs usage:

AI-Tools / LLMs that were used are:

• none

[AnnsAnns]

Is it ethical to approve this with "LGTM"

[Teufelchen1]

Looks like LLM slop. Just merge yolo

[benpicco]

Is it ethical to approve this with "LGTM"

Murdock will have the last word on that

[riot-ci]

Murdock results

❌ FAILED

0aa9603 fixup! treewide: mark functions as used to avoid LTO linker problems

Build failures (1)

Application	Target	Toolchain	Runtime (s)	Worker
tests/sys/malloc_monitor	atmega256rfr2-xpro	gnu	0.75	mobi6

Test failures (6)

Application	Target	Toolchain	Runtime (s)	Worker
tests/sys/malloc_monitor	native64	gnu	0.00	skyleaf
tests/sys/malloc_monitor	native32	gnu	0.00	mobi6
tests/sys/ps_schedstatistics	native32	gnu	0.00	mobi6
tests/sys/ps_schedstatistics	native64	gnu	0.00	mobi6
tests/sys/ssp	native32	gnu	0.00	mobi3
tests/sys/ssp	native64	gnu	0.00	mobi1

Artifacts

[crasbe]

Reddit says, that the llvm errors could be solved like this: https://www.reddit.com/r/Gentoo/comments/1e9j9et/comment/lefc7qi/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Adding -fuse-ld=lld to the LDFLAGS works, but that also requires to have lld installed.

Edit: I just checked and lld is already installed in our riotbuild container.

The LDFLAG addition of course has to be guarded to only be added when using LLVM.

[mguetschow]

Now it seems we are mostly down to this ESP32 (gcc) issue:

/opt/esp/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/lib/esp32/libg_nano.a(libc_a-__call_atexit.o):/builds/idf/crosstool-NG/.build/xtensa-esp-elf/src/newlib/newlib/libc/stdlib/__call_atexit.c:17:(.data+0x0): undefined reference to `__lock___atexit_recursive_mutex'
collect2: error: ld returned 1 exit status

and some spurious ones (non-exhaustive list):

/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/hard/libc_nano.a(lib_a-signal.o): in function `_raise_r':
signal.c:(.text._raise_r+0x2e): undefined reference to `_kill_r'
collect2: error: ld returned 1 exit status

Building application "gcoap_example" for "hifive1b" with CPU "fe310".
(...)
/opt/xpack-riscv-none-elf-gcc-13.2.0-2/bin/../lib/gcc/riscv-none-elf/13.2.0/../../../../riscv-none-elf/bin/ld: /tmp/ccbgYrQn.ltrans1.ltrans.o:(.text.trap_entry.lto_priv.0+0x34): undefined reference to `handle_trap'
collect2: error: ld returned 1 exit status

Building application "lwip_ipv4" for "nrf52840dk" with CPU "nrf52".
(...)
/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/hard/libc_nano.a(lib_a-signal.o): in function `_raise_r':
signal.c:(.text._raise_r+0x20): undefined reference to `_getpid_r'
/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: signal.c:(.text._raise_r+0x2e): undefined reference to `_kill_r'
collect2: error: ld returned 1 exit status

Building application "tests_nanocoap_cli" for "atmega256rfr2-xpro" with CPU "atmega256rfr2".
(...)
/tmp/ccm2IZTu.ltrans0.ltrans.o: In function `_cmd_put':
<artificial>:(.text._cmd_put+0xf6): undefined reference to `__wrap_strerror'
/tmp/ccm2IZTu.ltrans11.ltrans.o: In function `_nanocoap_put_handler':
<artificial>:(.text._nanocoap_put_handler+0x102): undefined reference to `__wrap_strerror'
/tmp/ccm2IZTu.ltrans11.ltrans.o: In function `_nanocoap_get_handler':
<artificial>:(.text._nanocoap_get_handler+0xa2): undefined reference to `__wrap_strerror'
<artificial>:(.text._nanocoap_get_handler+0x250): undefined reference to `__wrap_strerror'
collect2: error: ld returned 1 exit status

Building application "tests_ssp" for "nrf52840dk" with CPU "nrf52".
(...)
/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /tmp/tests_ssp.elf.bo40wB.ltrans0.ltrans.o: in function `uart_write.constprop.0':
<artificial>:(.text.uart_write.constprop.0+0x36): undefined reference to `__stack_chk_fail'
/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: <artificial>:(.text.uart_write.constprop.0+0x9c): undefined reference to `__stack_chk_guard'
collect2: error: ld returned 1 exit status

[mguetschow]

I was so sneaky as to add a commit that adds some __attribute__((used)). I have no clue why the linker is to dumb to find it out by itself, but marking it as used explicitly fixes the linking errors at least for me.

I've also double-checked that the functions actually do not end up in the binary when they are not actually used, and it appears they do not: The .text size of examples/basic/hello-world for nrf52840dk which uses newlib_nano by default did not change after I marked some newlib_syscalls_default functions as used. Also arm-none-eabi-nm -gC does not report the functions' names.

Let's see what else Murdock comes up with.

[benpicco]

I think the AVR GCC in riotdocker is a bit too vintage for LTO

/tmp/ccoUJiTf.s: Assembler messages:
/tmp/ccoUJiTf.s:1849: Error: register number above 15 required
lto-wrapper: fatal error: avr-gcc returned 1 exit status
compilation terminated.
/usr/lib/gcc/avr/5.4.0/../../../avr/bin/ld: error: lto-wrapper failed

[mguetschow]

I think the AVR GCC in riotdocker is a bit too vintage for LTO

But it might also be just an issue with that particular example, given that all the other binaries build just fine also for AVR.

[mguetschow]

Regarding the failing tests/sys/ssp:

$ make -C tests/sys/ssp all -j test                  
(...)
main(): This is RIOT! (Version: 2026.07-devel-300-g0aa960-use_lto)
calling stack corruption function
make[1]: *** [/home/mikolai/TUD/Code/RIOT-build/Makefile.include:877: cleanterm] Segmentation fault
Unexpected end of file in expect script at "child.expect('.*stack smashing detected.*')" (tests/sys/ssp/tests/01-run.py:15)

Process already stopped
make: *** [/home/mikolai/TUD/Code/RIOT-build/makefiles/tests/tests.inc.mk:32: test] Error 1
make: Leaving directory '/home/mikolai/TUD/Code/RIOT-build/tests/sys/ssp'

It seems from looking at the binary, that the stack protection is not added to the function by GCC when LTO is enabled, although the buffer is still allocated on the stack and written over its bounds (hence the segmentation fault). This contradicts the GCC doc on -fstack-protect and looks like a bug in GCC. I've found two possible workarounds to get GCC to still add stack protection to the function with LTO enabled:

1. Change -fstack-protector to -fstack-protector-strong. It seems to me that this is anyhow the more sensible option, especially given this is an opt-in feature.
2. Add __attribute__((stack_protect)) to test_func(). This attribute however is not recognized by clang (and hence would need to be guarded behind a define).

Any opinions on which approach we should take? Btw, found #13175 on the way which might be good giving another spin as well.