Add verify-only LMS and XMSS support

[Frauschi]

Summary

Adds PKCS#11 verify-only support for the two NIST-approved stateful hash-based signature (SHBS) families:

• LMS / HSS (RFC 8554, NIST SP 800-208)
• XMSS / XMSS^MT (RFC 8391, NIST SP 800-208)

Signature verification and raw public-key import are implemented on top of wolfSSL's wc_LmsKey_* / wc_XmssKey_* APIs, using the standard OASIS PKCS#11 v3.3 mechanism, key-type, and attribute constants. Everything is gated behind the new --enable-lms and --enable-xmss build options and is off by default.

This is step 1 (verify only). Signing, key generation, and private-key import are deliberately out of scope and are rejected at runtime; the object model and store layout are laid out so a future sign-capable step can add private keys without an interface break.

What's included

Interface (standard PKCS#11 v3.3)

Category	Constants
Key types	CKK_HSS (0x46), CKK_XMSS (0x47), CKK_XMSSMT (0x48)
Mechanisms	CKM_HSS (0x4033), CKM_XMSS (0x4036), CKM_XMSSMT (0x4037)
Attributes	CKA_HSS_LEVELS, CKA_HSS_LMS_TYPE(S), CKA_HSS_LMOTS_TYPE(S), CKA_HSS_KEYS_REMAINING, CKA_PARAMETER_SET (XMSS)

All values match the OASIS PKCS#11 v3.3 working headers exactly (not vendor-defined).

Behaviour

• Public-key import — a public key is an ordinary token or session object built from the raw RFC 8554 / RFC 8391 key in CKA_VALUE. wolfSSL derives the parameter set from the key bytes.
• Parameter validation — supplied CKA_HSS_* parameter attributes and the XMSS CKA_PARAMETER_SET OID are validated against the imported key; a mismatch, a malformed key, or a wrong-length key is rejected with CKR_ATTRIBUTE_VALUE_INVALID.
• Attribute read-back — the HSS parameter attributes and the XMSS CKA_PARAMETER_SET are exposed on read. CKA_HSS_KEYS_REMAINING is a private-key state property and is reported as unavailable in a verify-only build.
• C_Verify — validates the mechanism against the key type (CKR_KEY_TYPE_INCONSISTENT on a mismatch) and reports a wrong-length signature (both schemes) or an empty message (XMSS only, a wolfSSL backend limitation) as an invalid signature rather than a function failure.
• Persistence — public keys are stored and reload from disk via new store types WOLFPKCS11_STORE_HSSKEY_PUB (0x11) and WOLFPKCS11_STORE_XMSSKEY_PUB (0x13), with 0x10 / 0x12 reserved for future private keys.

Verify-only constraints

• Key generation (CKM_*_KEY_PAIR_GEN) and private-key import are rejected.
• Copying any HBS key is prohibited, so that a future sign-capable build can never duplicate one-time-signature state.

Build

# wolfSSL must be built with LMS and XMSS enabled:
./configure --enable-lms --enable-xmss ...
make && sudo make install

# wolfPKCS11:
./configure --enable-lms --enable-xmss
make && make check

Testing

New coverage in tests/pkcs11v3test.c and a dedicated tests/hbs_persistence_test.c:

• Known-answer verification for HSS, XMSS, and XMSS^MT (valid, tampered, and wrong-length signatures).
• Parameter-set validation (matching and mismatching CKA_HSS_* / CKA_PARAMETER_SET).
• Attribute read-back, including the per-level array attributes and the unavailable CKA_HSS_KEYS_REMAINING.
• Token-object persistence across a full library reload.
• Negative cases: private-key import rejected, key generation rejected, wrong mechanism/key-type pairings, empty templates, and malformed, wrong-length, and structurally invalid public keys.

All tests pass; the key-lifecycle paths (import, atomic re-import, free) were additionally checked clean under AddressSanitizer, including a --disable-rsa --enable-lms build.

Notes and limitations

• wolfSSL uses uniform LMS/LMOTS parameters across all HSS levels, so the per-level array attributes (CKA_HSS_LMS_TYPES / CKA_HSS_LMOTS_TYPES) report the single top-level type for every level. Heterogeneous HSS keys are not expressible today.
• The spec defines no XMSS "keys remaining" attribute; XMSS exposes only CKA_PARAMETER_SET.
• The CKL_LMS_* / CKL_LMOTS_* convenience macros are wolfPKCS11-local (the spec header defines none); their values are the RFC 8554 registry codes.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #206

Scan targets checked: wolfpkcs11-bugs, wolfpkcs11-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #206

Scan targets checked: wolfpkcs11-bugs, wolfpkcs11-src

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[Frauschi]

The failing CMake Build Test will be resolved once wolfSSL/wolfssl#10929 lands.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #206

Scan targets checked: wolfpkcs11-bugs, wolfpkcs11-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[philljj]

the CMake Build Tests is failing:

In file included from /home/runner/work/wolfPKCS11/wolfPKCS11/src/internal.c:76:
/home/runner/work/wolfPKCS11/wolfPKCS11/wolfpkcs11/internal.h:133:2: error: #error Compiling with XMSS requires XMSS support in wolfSSL (--enable-xmss).
  133 | #error Compiling with XMSS requires XMSS support in wolfSSL (--enable-xmss).
      |  ^~~~~

edit: is a separate issue: #206 (comment)

edit2: it passes now with that PR merged.

[philljj]

First pass. So far looks good, just some suggestions and comments.

[Copilot]

Pull request overview

Adds verify-only PKCS#11 v3.3 support for stateful hash-based signatures (HSS/LMS and XMSS/XMSS^MT) in wolfPKCS11, integrating wolfSSL’s LMS/XMSS APIs behind new build-time feature flags and extending the object store + tests/CI accordingly.

Changes:

• Introduces new PKCS#11 v3.3 key types/mechanisms/attributes for HSS and XMSS, plus verify-only mechanism wiring (C_VerifyInit/C_Verify).
• Implements raw public-key import, parameter-set handling, attribute read-back, and on-disk persistence for HSS/XMSS public keys.
• Adds comprehensive KAT + persistence tests and expands autotools/CMake/CI configurations for LMS/XMSS builds.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
wolfpkcs11/store.h	Adds new store type IDs for HSS/XMSS public keys.
wolfpkcs11/pkcs11.h	Defines PKCS#11 v3.3 HSS/XMSS constants and related typedefs/macros.
wolfpkcs11/internal.h	Adds wolfSSL LMS/XMSS includes, build guards, and internal APIs for import/verify.
tests/pkcs11v3test.c	Adds KAT tests + negative/attribute/guard coverage for HSS/XMSS verify-only behavior.
tests/include.am	Wires new persistence test + distributes hbs_kat.h.
tests/hbs_persistence_test.c	New test to ensure token public keys persist and still verify after reload.
tests/hbs_kat.h	Adds shared KAT vectors for HSS/XMSS/XMSSMT.
src/slot.c	Exposes new mechanisms and mechanism-info flags (verify-only).
src/internal.c	Implements HSS/XMSS key import, verify, attribute getters, and store encode/decode paths.
src/crypto.c	Integrates new key types into Create/SetAttributeValue and verify dispatch.
configure.ac	Adds --enable-lms / --enable-xmss configure flags and associated gating.
CMakeLists.txt	Adds CMake options for LMS/XMSS, plus conditional test enablement.
cmake/options.h.in	Exposes WOLFPKCS11_LMS / WOLFPKCS11_XMSS in generated options header.
.github/workflows/unit-test.yml	Adds new CI job matrices for LMS/XMSS builds.
.github/workflows/cmake.yml	Enables XMSS/LMS in wolfSSL build and LMS/XMSS in wolfPKCS11 CMake CI config.
.github/workflows/build-workflow.yml	Builds wolfSSL with LMS/XMSS enabled for workflow reuse.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.