Seeking feedback: Node.js support for provider-backed private keys

[panva]

Hello,

I'm working on a Node.js PR to support OpenSSL provider-backed private keys in node:crypto, with pkcs11-provider as a realistic test case for non-exportable keys whose operations are performed by a token/HSM which we can execute in CI.

Node PR: nodejs/node#63949

The intended Node behavior is roughly:

• allow private-key inputs to be loaded from OpenSSL STORE URLs such as pkcs11:object=...;type=private;
• keep the private key non-exportable when the provider/token does not expose key material;
• still support operations such as sign/verify, public-key derivation, and WebCrypto CryptoKey wrapping where OpenSSL/provider semantics allow it;
• avoid hard-coding assumptions that are only true for one provider or one SoftHSM setup.

We have a local SoftHSM + pkcs11-provider harness covering RSA, RSA-PSS, EC P-256, Ed25519, Ed448, export behavior, WebCrypto wrapping, and some cipher/KEM/ECDH cases. Some things look straightforward, but others are harder to classify as "Node bug", "provider limitation", "token/mechanism limitation", or "expected OpenSSL provider contract".

Current areas where feedback would be especially useful:

1. Which operations should a consumer reasonably expect from PKCS#11 private keys loaded through STORE?
   For example: digest sign vs raw/prehashed sign, RSA-PSS parameters, EdDSA one-shot signing, private decrypt, ECDH, and KEM/decapsulate.

2. Are failures for private-key export, RSA private decrypt, EC ECDH, or KEM operations expected with the current SoftHSM + pkcs11-provider setup, or are there provider parameters/mechanisms Node should be using differently?

3. For ECDSA, Node can sometimes recover by using the lower-level prehashed sign/verify path when provider digest-sign is unavailable. Does that match how pkcs11-provider expects consumers to use ECDSA keys, or is there a better EVP-level pattern?

4. Are there additional key types or operations we should include in the harness?
   SM2, DSA, X25519/X448, and eventually PQC keys are all areas where we do not want to make incorrect assumptions from a narrow RSA/EC test matrix.

5. Would anyone with access to real tokens/HSMs be willing to try the Node branch and report which operations work or fail?

I'm not asking pkcs11-provider to conform to Node's assumptions. The goal is the opposite: make sure Node's API behavior is aligned with OpenSSL provider semantics and tested against realistic provider-backed keys.

Even short guidance like "this failure is expected", "you should call this EVP operation instead", or "SoftHSM cannot represent this mechanism usefully" would help a lot.

[Jakuje]

This sounds to me more like a discussion thread than the issue, but I will leave it up to Simo to move it.

Which operations should a consumer reasonably expect from PKCS#11 private keys loaded through STORE?
For example: digest sign vs raw/prehashed sign, RSA-PSS parameters, EdDSA one-shot signing, private decrypt, ECDH, and KEM/decapsulate.

All these sounds reasonable. I would add wrapping/unwrapping, which is a common use case for key management in HSMs, but not sure how much it is now supported in pkcs11-provider.

Are failures for private-key export, [...] expected with the current SoftHSM + pkcs11-provider setup, or are there provider parameters/mechanisms Node should be using differently?

Yes. The main point of HSM is that you do not export the private key. It will fail.

private decrypt, EC ECDH, or KEM operations [...]

these should work.
We do not test KEM operations with SoftHSM as there is not yet a public release supporting this. The pkcs11-provider is tested against kryoptic, where we know these operations work.

For ECDSA, Node can sometimes recover by using the lower-level prehashed sign/verify path when provider digest-sign is unavailable.

The SoftHSM has bugs, including softhsm/SoftHSMv2#842, causing some of the hashed ECDSA operations not working. This is a known bug, but this is a bug you can expect also in physical HSMs (but less likely). We have a workaround in the pkcs11-provider tests which disable these in softhsm and fall back to the raw ECDSA in the pkcs11-provider level:

https://github.com/openssl-projects/pkcs11-provider/blob/main/tests/softhsm-init.sh#L56

Does that match how pkcs11-provider expects consumers to use ECDSA keys, or is there a better EVP-level pattern?

It depends on what is the aim of your API. Generally, the EVP_PKEY API is the one that should be used going forward. Using other low-level APIs will go away in the future OpenSSL versions and will just cause more headache in the long run. The EVP_PKEY API is also useful if you aim for some certification compatibility, which requires the compound operation to be done inside the module, rather than broken to separate steps.

Are there additional key types or operations we should include in the harness?
SM2, DSA, X25519/X448, and eventually PQC keys are all areas where we do not want to make incorrect assumptions from a narrow RSA/EC test matrix.

There is no support for SM2 nor DSA in pkcs11-provider.

EDDSA signatures should be included as they are part of many signature schemes these days. The softHSM has limited suport for these (see softhsm/SoftHSMv2#873), but from the OpenSSL layer, using these should be mostly transparent to what you do with RSA or ECDSA.

PQC signature algorithms should also be included in matrix eventually. We test these in pkcs11-provider with kryoptic. The usage through EVP_PKEY API should be also transparent, but if there are some fixed size buffers, you might hit them as either signatures or keys are much larger.

Would anyone with access to real tokens/HSMs be willing to try the Node branch and report which operations work or fail?

I do not have any HSM to play with to try this. Adding test with different software tokens as we have in pkcs11-provider testsuite is probably as much as I can recommend.