Bound POSIX SHM mapping by the shared object size

[yosuke-wolfssl]

Problem

The POSIX shared-memory transport built its mmap layout from header fields that live in the shared object itself, so a peer could set them freely. A local process that wins the race to create the named object (mode 0660, same uid/gid) could:

• Wrap the length. size = sizeof(header) + req_size + resp_size + dma_size had no overflow check and was never compared against the object's real size. The sum could wrap to a small value; Map then mapped that small region but derived resp/dma from the raw fields, placing them outside the mapping, and propagated a near-SIZE_MAX dma_size that defeated the client DMA bounds check.
• Overflow isn't even required — req_size = resp_size = 65535 on a 64-byte object gives a non-wrapping size past EOF; touching the last page is SIGBUS.
• Rewrite between validation and use. The header was read twice — validated from a probe mapping, then re-read from a fresh mapping to place the buffers — so a hostile peer could change the fields in between.

Addressed by f_4223.

Fix (port/posix/posix_transport_shm.c)

• fstat is the authoritative bound. posixTransportShm_Map derives the mapping length from the object size, not the header.
• Snapshot-then-validate. req_size/resp_size/dma_size are read once into locals, checked (sizeof(header) + req + resp <= st_size, dma_size <= remainder, overflow-free), and used only from those locals. HandleMap and ServerInit consume the validated map->req_size/map->resp_size instead of re-reading the header.
• Buffer sizing rule. Each buffer must be a non-zero multiple of sizeof(whTransportMemCsr). A buffer under one word underflowed the length check in wh_TransportMem_SendRequest (req_size is uint16_t, so req_size - sizeof(csr) wrapped near SIZE_MAX), leaving an out-of-bounds write reachable; a non-multiple misaligned the response/DMA pointers cast to volatile uint64_t.
• Server fails fast. CreateMap applies the same rule to the config before shm_open, so a misconfigured server returns WH_ERROR_BADARGS instead of leaving a stale object every client would reject; it also unlinks if the post-create map fails. Documented on posixTransportShmConfig in the header.

Tests (test/wh_test_comm.c)

New single-process negative tests under whTest_Comm. _whTest_CommShmMalformedHeader squats the name and presents 8 malformed layouts plus one exactly-fitting accept case; _whTest_CommShmServerBadConfig drives ServerInit with a bad size and asserts no object is left linked. Each clause of the predicate and both sides of the DMA bound have coverage.

Verification

• Build clean under -std=c90 -Werror -Wall -Wextra -Wpointer-arith.
• Legacy suite and test-refactor (37 passed, 0 failed) pass; POSIX example pair connects and runs its full crypto sequence.
• Mutation-checked: dropping any clause of the new validation, flipping the DMA bound >→>= or >→>+1, drifting the test's header mirror, or removing the server config guard each makes the suite fail — the guards are load-bearing, not decorative.

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR hardens the POSIX shared-memory transport against malicious or malformed SHM headers by bounding mappings to the actual SHM object size, snapshotting and validating peer-writable header fields once, and enforcing alignment/size rules for request/response buffers. It also adds regression tests that squat the SHM name and validate both client-side rejection and server-side fail-fast behavior.

Changes:

• Derive mmap length from fstat() object size and validate a single snapshot of header sizes before computing buffer offsets.
• Enforce “non-zero multiple of sizeof(whTransportMemCsr)” for request/response buffer sizes on both client and server creation paths.
• Add negative tests for malformed SHM headers and for server misconfiguration leaving no stale SHM objects behind.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
test/wh_test_comm.c	Adds negative SHM layout/config tests by squatting the SHM name and writing malformed headers.
port/posix/posix_transport_shm.h	Documents the new req/resp buffer sizing/alignment rule and expected error behavior.
port/posix/posix_transport_shm.c	Maps SHM based on fstat() size, snapshots + validates header fields once, and uses validated sizes throughout.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #479

Scan targets checked: wolfhsm-core-bugs, wolfhsm-src

No new issues found in the changed files. ✅

[rizlik]

The PR improves the POSIX SHM robustness.
What I see problematic is that it might hint that the POSIX SHM can provide a true isolation boundary between the client and the server.

Unfortunately there are plenty of other attack vectors for the client over POSIX SHM (both peers typically run with same UID, a malicious client can still ftruncate the file and SIGBUS the server, and an attacker can race to create a same-name object and interpose between client and server).

Moreover, I don't think POSIX SHM was ever meant to be a secure transport protocol but merely used for test, debug and as a reference.

I'm OK with the PR but this is the right moment to document the threat model explicitly in posix_transport_shm.h: same trust domain assumed; no peer authentication or isolation;

[rizlik]

NIT: superfluous

[yosuke-wolfssl]

Dropped, it's reverted to the original line.

[rizlik]

superfluous

[yosuke-wolfssl]

Same here, reverted

[yosuke-wolfssl]

Hi @rizlik ,
Thank you for reviewing.
I agreed with your comment on thread model — the framing was the real problem with the PR. POSIX SHM is a test/debug/reference transport and nothing here changes that; a peer at the same uid can still ftruncate() the object and fault the other side, or win the name race and interpose.

Added an explicit Threat model section to the top of posix_transport_shm.h stating: same trust domain assumed (same uid, peer contents are trusted input), no peer authentication (whoever opens the name is the peer, and the client unlinks on successful mapping so the name can be recreated by anyone afterwards), and no isolation or availability guarantee (ftruncate(), or simply never advancing the CSRs).

It closes by scoping what this PR actually buys: the header fields are validated against the real object size so a malformed or truncated object can't drive an out-of-bounds access in this process — robustness against a corrupt object, not protection against a hostile peer. Explicitly not a promotion to a trusted channel.