Fix: WolfCrypt 7 fenrir fixes

[aidankeefe2022]

Description

Fixed a series of Fenrir issues that mostly pertained to alignment UB when reading byte arrays as various sizes of integers.

https://fenrir.wolfssl.com/finding/6438

FSPSM RSA private-decrypt never updates caller's *outLen (passes &outLen instead of outLen)

• Swapped the double reference to just pass the outLen ptr

https://fenrir.wolfssl.com/finding/5414

SLH-DSA AVX2 x4 SHAKE helpers cast hash byte streams to word64

• Changed from direct cast byte -> word64 / word64 -> byte to use Unaligned memory helpers

https://fenrir.wolfssl.com/finding/5415

ML-DSA little-endian pack/unpack paths cast encoded byte buffers to word pointers

• Fixed a variety of possible alignment issues.

https://fenrir.wolfssl.com/finding/6165

scrypt scryptROMix big-endian Integerify shifts bytes without width cast (signed overflow/oversized shift on 16-bit int)

• Explicitly cast all intermediate values to word32 to make sure that there is no chance of signed in overflow during bit shifting.

https://fenrir.wolfssl.com/finding/6166

wc_ecc_init_id (SE050) reinterprets byte id[] array as word32 via pointer cast

• Used unaligned helper function to read byte array as word32

https://fenrir.wolfssl.com/finding/6167

XMSS hash-address byte buffers cast to word32* in address-encoding helpers (unaligned access / strict aliasing)

• Removed cast to use endian independent operation that copies the byte value in to the last byte because the word is stored in network byte order

https://fenrir.wolfssl.com/finding/6177

PKCS#12 parse leaves decrypted SafeContents (incl. plaintext private KeyBag) in freed heap buffer

• Added new variable to track temp buffer size and ForceZero it when it is freed

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10808

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-src

No new issues found in the changed files. ✅

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10808

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-src

No new issues found in the changed files. ✅

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10808

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-src

No new issues found in the changed files. ✅

[Frauschi]

Jenkins retest this please

[Frauschi]

LGTM

[aidankeefe2022]

retest this please

[aidankeefe2022]

retest this please

[Copilot]

Pull request overview

This PR addresses multiple Fenrir findings in wolfCrypt by removing undefined behavior caused by unaligned/aliasing-unsafe pointer casts when reading/writing integer types from byte buffers, and by fixing an output-length propagation bug in the Renesas FSPSM RSA private decrypt path.

Changes:

• Reworked XMSS address byte-setting to use byte-wise stores instead of casting encoded address buffers to word32*.
• Updated SLH-DSA AVX2 x4 SHAKE helpers to use readUnalignedWord64 / writeUnalignedWord64 instead of casting hash buffers to word64*.
• Fixed scrypt big-endian Integerify shifts to avoid integer-promotion/shift UB, fixed SE050 ECC key-id reads using unaligned helpers, and corrected FSPSM RSA private decrypt to update the caller’s *outLen.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
wolfcrypt/src/wc_xmss_impl.c	Removes word32* casting from encoded XMSS address manipulation to avoid unaligned access / strict-aliasing UB.
wolfcrypt/src/wc_slhdsa.c	Replaces word64* casts in AVX2 x4 SHAKE hash buffer helpers with unaligned read/write helpers.
wolfcrypt/src/pwdbased.c	Adds explicit word32 casts in scrypt big-endian Integerify composition to prevent shift/promotion UB.
wolfcrypt/src/port/Renesas/renesas_fspsm_rsa.c	Fixes RSA private decrypt to pass the correct outLen pointer so the caller’s *outLen is updated.
wolfcrypt/src/ecc.c	Uses readUnalignedWord32 to safely read SE050 key IDs from key->id without alignment/aliasing UB.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.