bsdkm x86_vecreg: nest counter, and other improvements.

[philljj]

Description

Adding a nesting counter max for fpu state management in freebsdkm.

Testing

[Copilot]

Pull request overview

This PR updates the FreeBSD kernel-module vector/FPU state management helpers to better guard against runaway save/restore nesting and to slightly optimize error-path branching.

Changes:

• Add WOLFKMOD_FPU_MAX_NEST and enforce an upper bound on fpu_states[].nest during nested wolfkmod_vecreg_save() calls.
• Add __predict_false(...) annotations to several error checks to bias branch prediction toward the success path.

Comments suppressed due to low confidence (1)

bsdkm/x86_vecreg.c:197

• On the error path after calling wolfkmod_fpu_kern_enter(), the function returns EINVAL without calling wolfkmod_fpu_kern_leave(). That can leave PCB_KERNFPU set and keep migration/preemption restrictions in effect for the current thread, potentially causing hard-to-debug scheduler/FPU state issues.

        if (__predict_false(fpu_states[PCPU_GET(cpuid)].nest != 0 ||
            td_tid != 0)) {
            printf("error: wolfkmod_fpu_kern_enter() with nest: %d, %d\n",
                   fpu_states[PCPU_GET(cpuid)].nest, td_tid);
            return (EINVAL);
        }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[philljj]

I think it's fine. The point is we're over that high water mark which is very bad.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[philljj]

Retest this please.

java.io.IOException: Unexpected EOF.