[cpullvm] - Prepare for LIBCXX tests enablement ARM/RISC-V.

[Pawan Nirpal (pawan-nirpal-031)]

Manage fixes and split testing script to run check-cxx separately to reduce the testing time for libcxx tests.

[Pawan Nirpal (pawan-nirpal-031)]

Jonathon Penix (@jonathonpenix) Could you please review and add other reviewers as you see fit.

[Jonathon Penix (jonathonpenix)]

At a glance, it looks like this is headed in the right direction, thanks Pawan!

Few misc. thoughts:

The libc++abi/libunwind tests are also included under the broader "enable libc++ tests". Right now they're running as we just run all enabled tests:

cpullvm-toolchain/qualcomm-software/scripts/test.sh

Lines 20 to 24 in 8f6aeba

	# Run all relevant test targets. This might be too broad eventually,
	# but while we have a limited number of variants (and no compiler-rt
	# or libc++ testing enabled) we can run everything.
	cd "${REPO_ROOT}"/build
	ninja check-all-llvm-toolchain

I'm fine if we enable this stuff incrementally, but I think we have a few options to do so. Things that make the most sense to me:

1. If we get down to just test failures, we can xfail and gradually re-enable tests and retain the check-all*
2. If we want to ex: merge this and address libcxxabi/libunwind build failures later, we can break up the check-all-llvm-toolchain into smaller components and exclude abi/unwind.

Pawan Nirpal (@pawan-nirpal-031) do you know/remember how long the libc++ tests were taking to run on your local machine?

Arm has these split out as they apparently take a long time to run, so we might want to do # 2 above even if we resolve build/test failures.

[Sam Elliott (lenary)]

I am not 100% sure how this ends up working, but are we correctly setting long_tests and large_tests? I think we should be if we're using qualcomm-software/embedded-runtimes/test-support/llvm-libc++-picolibc.cfg.in but I'm not sure if we are using that config or something else.

These prevent some of the really slow tests from running.

[Ana Pazos (apazos)]

They are already turned off in
embedded-runtimes/test-support/llvm-libc++-picolibc.cfg.inllvm-libc++-picolibc.cfg.in:
config.long_tests = False
config.large_tests = False

Pawan Nirpal (@pawan-nirpal-031), let's focus on fixing the issues first. We can merge the fixes without enabling the tests. Before activating the tests, we may need to split them into separate runs, as they seem to still take considerable time even with long/large tests disabled.

[Pawan Nirpal (pawan-nirpal-031)]

Ana Pazos (@apazos) Yes I think that would be preferable. I've fixed the tests, some of them have to be unsupported due to memory limitations set to 2MB IIRC, those tests run into memory issues and I suppose aren't meant for embedded. Long double tests have to be xfailed through and through. But I will update this PR and we can discuss on the approach more.

CC Jonathon Penix (@jonathonpenix) Sam Elliott (@lenary)

[Pawan Nirpal (pawan-nirpal-031)]

Large test

These tests allocate very large amounts of memory (1,000,000 element containers, 1M-element vectors) that exceed what's available in the QEMU semihosting environment for bare-metal targets. The process aborts when malloc fails or the heap overflows.
Specifically:
- flat.map/size.pass.cpp: inserts 1,000,000 elements into a flat_map
- sort.heap/complexity.pass.cpp: allocates a 1M-element vector (1 << 20) and runs std::mt19937 over it
- eval_param.pass.cpp and simd_copy.pass.cpp: similarly large workloads
The bare-metal QEMU machine has a fixed, limited RAM (defined by the linker script picolibcpp.ld). When the test tries to allocate more than what's available, malloc returns null or the heap collides with the stack, causing an abort (signal 6 → exit 134).

std/algorithms/alg.sorting/alg.heap.operations/sort.heap/complexity.pass.cpp
 libc++-aarch64a_tlsie :: std/algorithms/alg.sorting/alg.heap.operations/sort.heap/ranges_sort_heap.pass.cpp
  libc++-aarch64a_tlsie :: std/containers/container.adaptors/flat.map/flat.map.capacity/size.pass.cpp
  libc++-aarch64a_tlsie :: std/containers/container.adaptors/flat.set/flat.set.capacity/size.pass.cpp
  libcxx/test/std/experimental/simd/simd.class/simd_copy.pass.cpp

UNSUPPORTED : Picolibc doesn't yet support stdio for long doubles for aarch64/arm/riscv

std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/long_double.pass.cpp
		  libc++-aarch64a_tlsie :: std/language.support/support.start.term/quick_exit.pass.cpp
		  libc++-aarch64a_tlsie :: std/localization/locale.categories/category.monetary/locale.money.get/locale.money.get.members/get_long_double_overlong.pass.cpp
		  libc++-aarch64a_tlsie :: std/localization/locale.categories/category.numeric/locale.nm.put/facet.num.put.members/put_long_double.hex.pass.cpp
		  libc++-aarch64a_tlsie :: std/localization/locale.categories/category.numeric/locale.nm.put/facet.num.put.members/put_long_double.pass.cpp
		std/strings/string.conversions/to_string.pass.cpp
		std/strings/string.conversions/stold.pass.cpp
                

Picolibc simply doesn't implement quick_exit or at_quick_exit yet. 
libcxx/test/std/language.support/support.start.term/quick_exit.pass.cpp

yet to get a hold on this one : libcxx/test/std/strings/c.strings/no_c8rtomb_mbrtoc8.verify.cpp

[Ana Pazos (apazos)]

Hi Pawan Nirpal (@pawan-nirpal-031),

Besides tests failing, there are several ways to disable tests to control the time it takes to run the tests. Bringing them up here so we select the most appropriate one depending on the tests running time and the tests failing, so we can progressively enable them as we fix the issues:

1. Select the ninja check command to run in /qualcomm-software/qualcomm-software/scripts/test.sh
   Right now, we enable ninja check-all-llvm-toolchain (checks all LLVM/clang binaries and runtimes but excludes lib C++/abi/undwind tests)

We can consider enabling tests for libc++ in parts, as done in ATfe, they skip ninja check-cxx which takes long, see ATfE /embedded/scripts/test.sh
ninja check-all (checks all LLVM/clang binaries and runtimes including compiler-rt, lib C++/abi/undwind)
ninja check-all-llvm-toolchain (excludes lib C++ runtimes tests)
ninja check-cxxabi
ninja check-unwind
ninja check-cxx

2. ATfE has a separate /embedded/scripts/test_libcxx.sh where they only enable ninja check-cxx

3. This alternative allows to specify the particular project, variants and tests to disable
   https://github.com/qualcomm/cpullvm-toolchain/blob/qualcomm-software/qualcomm-software/embedded-runtimes/test-support/xfails.py
   We can check the tests disabled in ATfE /embedded/arm-runtimes/test_support/xfails.py

For example, if you only enable ninja check-cxxabi or ninja check-unwind or both, are there failures?

[Pawan Nirpal (pawan-nirpal-031)]

Ana Pazos (@apazos) Thanks for the detailed description, Yes I feel incrementally enabling would be the best option, ninja check-cxx are the ones that I am focusing on this PR. I imagine we don't want any source changes apart from changes in qualcomm-sotfware/* is that right? If yes I don't think there's much we can do to disable failing tests that need source changes, I'm wrestling with some riscv32* tests that are need support from upstream, Do you think we can push some of these libcxx source changes upstream first? Or we can disable the tests now and then start upstreaming fixes into libcxx?

[Ana Pazos (apazos)]

We have no custom LLVM changes in qualcomm-software and it keeps up with latest LLVM tip.
Which issue are you running into with riscv32* tests?
I am not sure the changes to tests are upstreamable (they reflect that in the particular runners/builders we are using they take a long time) and the way I see to disable the tests is along what ATfE does in /embedded/arm-runtimes/test_support/xfails.py.

[Pawan Nirpal (pawan-nirpal-031)]

Ana Pazos (@apazos) Jonathon Penix (@jonathonpenix) Sam Elliott (@lenary) I have worked libcxx into a different test_libcxx.sh test script file similar to ATFE, All the tests locally pass, however I am omitting check unwind, abi and picolibc tests. The ETA for the libcxx tests locally is around 6-6:30 hours. The failure now seems to be purely CI timeout. The job has exceeded the maximum execution time of 6h0m0s

[Ana Pazos (apazos)]

Pawan Nirpal (@pawan-nirpal-031) this patch will have to be further cleaned up as some of these changes you are pushing in a separate PR. Also, might make sense to combine the xfailed test patch with the patch that enables the tests.

[Pawan Nirpal (pawan-nirpal-031)]

Pawan Nirpal (@pawan-nirpal-031) this patch will have to be further cleaned up as some of these changes you are pushing in a separate PR. Also, might make sense to combine the xfailed test patch with the patch that enables the tests.

Hi Ana Pazos (@apazos) I'm sorry I didn't quite understand, you said we were only going to push the fixes and defer the enablement to a later PR. I will clean up the infra changes alright. Then as per me this PR should have xfails.py (fixes) and the separation of test_libcxx.sh. So if we also push the xfails to the enablement patch, this is just a CI change patch which separates the ninja check, is that what we intend to achieve? Apologies for the confusion.

[Pawan Nirpal (pawan-nirpal-031)]

Ana Pazos (@apazos) Jonathon Penix (@jonathonpenix) could you review it now. Also this seems to be failing the license check what would be the appropriate one?

[Jonathon Penix (jonathonpenix)]

This looks pretty close to me, thanks a lot Pawan Nirpal (@pawan-nirpal-031) for working on this.

Also this seems to be failing the license check what would be the appropriate one?

Apache 2 + LLVM is right, AFAIK that check assumes BSD 3 clause by default which is where our errors generally come from--it's something we have to sort out with the folks who maintain that. So, there's nothing you need to change here AFAIK

[Jonathon Penix (jonathonpenix)]

Is this test not running for RISC-V or is it actually passing?

Seems surprising to me that we wouldn't have to handle RISC-V variants similarly. (I assume RISC-V QEMU isn't that much faster than Arm/AArch64?)

[Jonathon Penix (jonathonpenix)]

Pawan Nirpal (@pawan-nirpal-031) it seems these xfails were removed entirely in the latest patch? Does that mean all targets are passing without these xfails?

[Jonathon Penix (jonathonpenix)]

Do you have a rough idea how long the libc++ tests for an individual variant take to run? Can you check on this?

Needs input from others, but it might be worthwhile including at least a couple "core" variants to be tested during premerge (and normal nightly?) checks and we can run the full suite (like what you did for test_libcxx.sh) separately. See ex: https://github.com/arm/arm-toolchain/blob/arm-software/arm-software/embedded/scripts/test_pre_merge.sh

[Jonathon Penix (jonathonpenix)]

Ping

[Jonathon Penix (jonathonpenix)]

We might want to omit this part for now.

I think eventually we want something like this (we've discussed re-adding this into test.sh a few times as we've already been bitten a few times by the "exit at first fail" hiding other issues) but we'd need people to be aware to look at this (or, do some more handling of it to make the failures more obvious).

Ana Pazos (@apazos) do you have thoughts/preferences here?

(I guess this is also a bit moot though until we actually schedule something to run these tests)

[Pawan Nirpal (pawan-nirpal-031)]

Pawan Nirpal (@pawan-nirpal-031) it seems these xfails were removed entirely in the latest patch? Does that mean all targets are passing without these xfails?

Jonathon Penix (@jonathonpenix) these tests take a long time to run, I wanted to xfail them to investigate later, they take long enough that they don't report a pass or a fail. But I'm skeptical if CI will actually allow them to run that long. So I've removed them for now.

[Jonathon Penix (jonathonpenix)]

This isn't right--there are variants that are specifically/intentionally built under initial-exec to support non-static executables. We can't just not do that anymore.

If you're seeing an issue, please provide more information.

The __tls_get_addr issue described in the comment should already have been addressed (that's what the lines that were deleted are there for).

[Garvit Gupta (quic-garvgupt)]

There are 7 variants for which libc testing is enabled but not libcxx tests.

• armv7a_hard_vfpv3
• armv7m_hard_fpv5_d16_nopic
• riscv32imc_ilp32_nothreads_nopic
• riscv32imc_zba_zbb_zbc_zbs_ilp32_nothreads_nopic
• riscv32im_xqci_ilp32_nothreads_nopic
• riscv32im_zba_zbb_zbc_zbs_zca_zcb_zcmp_ilp32_nothreads_nopic
• riscv64imc_lp64_nothreads_nopic

Pls enable libcxx tests for the above variants as well.

[Garvit Gupta (quic-garvgupt)]

There are 7 variants for which libc testing is enabled but not libcxx tests.

• armv7a_hard_vfpv3
• armv7m_hard_fpv5_d16_nopic
• riscv32imc_ilp32_nothreads_nopic
• riscv32imc_zba_zbb_zbc_zbs_ilp32_nothreads_nopic
• riscv32im_xqci_ilp32_nothreads_nopic
• riscv32im_zba_zbb_zbc_zbs_zca_zcb_zcmp_ilp32_nothreads_nopic
• riscv64imc_lp64_nothreads_nopic

Pls enable libcxx tests for the above variants as well.

FYI: These variants also has ENABLE_CXX_LIBS set to ON.

[Pawan Nirpal (pawan-nirpal-031)]

There are 7 variants for which libc testing is enabled but not libcxx tests.

• armv7a_hard_vfpv3
• armv7m_hard_fpv5_d16_nopic
• riscv32imc_ilp32_nothreads_nopic
• riscv32imc_zba_zbb_zbc_zbs_ilp32_nothreads_nopic
• riscv32im_xqci_ilp32_nothreads_nopic
• riscv32im_zba_zbb_zbc_zbs_zca_zcb_zcmp_ilp32_nothreads_nopic
• riscv64imc_lp64_nothreads_nopic

Pls enable libcxx tests for the above variants as well.

FYI: These variants also has ENABLE_CXX_LIBS set to ON.

Noted, added the two armv7s, will add riscv in the following patches due to CI timeout all versions cannot be accommodated.