pkcs11: getSession() can open unbounded HSM sessions under concurrent load

[EvanYan1024]

Problem

Provider.getSession() in bccsp/pkcs11/pkcs11.go falls back to createSession() whenever the cached session pool is empty, with no upper bound on the number of concurrently open PKCS#11 sessions:

func (csp *Provider) getSession() (session pkcs11.SessionHandle, err error) {
    for {
        select {
        case session = <-csp.sessPool:
            return
        default:
            // cache is empty (or completely in use), create a new session
            return csp.createSession()
        }
    }
}

Under high-concurrency signing workloads, the cache empties quickly and every caller takes the default branch, so the number of open sessions grows with the number of in-flight callers. The configured sessionCacheSize (default 10) only bounds how many idle sessions are cached in sessPool; it does not bound how many sessions can be open at once.

Impact

When the number of concurrent signers exceeds what the HSM / PKCS#11 token can support:

• The PKCS#11 library returns CKR_SESSION_COUNT from C_OpenSession, surfaced as OpenSession failed, retrying [pkcs11: 0xB1: CKR_SESSION_COUNT].
• Tokens may then return CKR_DEVICE_ERROR from C_Sign because session state on the device becomes exhausted/inconsistent.
• Callers see signing errors propagate up; in a Fabric deployment this surfaces as endorsement / envelope signing failures.

Reproduced on a deployment backed by a remote PKCS#11 KMS:

• Workload: ~800 concurrent sign callers across multiple FSC processes.
• Over a 2-minute window:
  • error submitting signature: ~2000
  • CKR_DEVICE_ERROR: ~5000 log lines / ~1574 unique tx
  • CKR_SESSION_COUNT: ~140 log lines
• Submit failure rate: ~3.7%.

Why sessionCacheSize alone is not enough

sessionCacheSize bounds the cache (sessPool), not the concurrency:

• Hot path: cache empty → createSession() → no gate.
• returnSession() only chooses between caching or closing on the return side; it cannot prevent unbounded OpenSession calls on the borrow side.

Proposed fix

Introduce a session-slot semaphore (sessionSlots) with the same capacity as sessPool, and gate createSession() behind it inside getSession(). Surface configuration via a PKCS11_SESSION_CACHE_SIZE environment variable so deployments can tune it without recompiling.

See #49 for the implementation. After deploying that PR with PKCS11_SESSION_CACHE_SIZE=30 against the same workload:

• error submitting signature: 0
• CKR_DEVICE_ERROR: 0
• CKR_SESSION_COUNT: 0
• Sign-phase p50 latency rose from ~17ms to ~48ms (controlled queueing under saturation, expected trade-off).

Environment

• fabric-lib-go: v1.1.3
• Go: 1.25
• PKCS#11 backend: remote KMS via PKCS#11 shim

Related

• Limit PKCS11 session creation #49