Add CI that builds and runs every example by aidangarske · Pull Request #598 · wolfSSL/wolfssl-examples

aidangarske · 2026-07-15T00:38:18Z

Adds CI that builds and runs every example against wolfSSL master and the latest stable tag, asserting on real output — not just exit 0 — and fixes the example bugs that blocked those runs from going green.

Issue-creation triage tested here: CI: hash failing in the nightly aidangarske/wolfssl-examples#2
Manifest coverage: 72 run, 37 build-only, 34 skipped (each with a reason) — 60 of the runs assert on output, not just exit 0.
History is one atomic commit per fix, so this can be reviewed fix-by-fix.

CI

Manifest-driven harness (run_example.py + examples-manifest.yml): build → run → assert expected output.
Per-target workflows for environments that need special toolchains: ESP32, Android, RT1060, UEFI, SGX, PSA/mbedTLS, device sims (SE050, ATECC608, STSAFE, TROPIC01, STM32, PIC32MZ), DTLS multicast, Windows/MSVC, cross-arch (m32/aarch64/s390x), sanitizers/valgrind/cppcheck, config-matrix.
Nightly runs everything vs master + stable and triages failures into per-directory GitHub issues (retry-once, then flake-vs-real classification).
Every network fetch (git clone, download, docker pull) is wrapped in a retry so a transient blip does not fail a job.

Fixes

Each is its own commit:

Expired fixtures — regenerated CRL + certs that silently broke the CRL/verify examples.
Wrong exit codes — servers/examples that "passed" by returning a byte count or an unconditional -1; they now fail when they should.
Never-worked examples — PQ/ML-DSA/ML-KEM API ports, a client-cert-less resume, identical XTS key halves.
Toolchain drift — ESP-IDF 5.x ENC28J60 + PHY ports, RT1060 cross-build, UEFI freestanding, Android NDK/API-31, mynewt 64-bit.
Real bugs — a custom-IO double-free/use-after-free, DTLS resume-on-close_notify + record-spanning, TLS 1.3 early-data session-ticket handling.

Upstream wolfSSL bugs this CI surfaced

Examples are marked known-fail and gated per ref with fixed_on; drop the marker when each merges:

OCSP stapling — a CTX-set status_request is dropped from the TLS 1.3 Certificate message (-429): Check ctx extensions before rejecting a TLS1.3 certificate message extension wolfssl#10936 (merged to master)
PKCS#11 — wc_ecc_verify_hash on a label-generated EC key returns WC_HW_E (-248): Set label and ID on PKCS11 EC public key wolfssl#10954 (in review)
PKCS#7 — smime/indef flags: Fix PKCS7 streamed SignedData dropping content wolfssl#10904
point to ocsp and pkcs back to master after wolfssl merges

Copilot

Pull request overview

Adds a comprehensive GitHub Actions CI system to build and (where possible) run every example in this repository against wolfSSL master and the latest stable tag(s), using a manifest-driven matrix so failures are isolated to per-example jobs.

Changes:

Introduces manifest-driven CI workflows (host/emulated/cross) plus a nightly triage workflow that retries, classifies flakes vs real failures, and can file/update issues.
Adds Python harness/scripts to generate matrices, enforce “coverage” (no buildable example directory left unaccounted for), run examples with output assertions, and collect artifacts for triage.
Updates many examples/Makefiles/scripts/fixtures so they build/run reliably under CI (exit statuses, paths to certs, linking flags, regenerated CRL/certs, etc.).

Reviewed changes

Copilot reviewed 62 out of 64 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
X9.146/Makefile	Makefile header tweak for X9.146 examples
x509_acert/README.md	Fix build/run instructions and cert paths
x509_acert/Makefile	Make OpenSSL prefix optional; adjust lib paths
uefi-static/Makefile	Make libgcc/gnu-efi paths less distro-specific
tls/server-tls13.c	Ensure successful exit status after shutdown
tls/server-tls13-certauth-clienthello.c	Ensure successful exit status after shutdown
tls/server-tls-posthsauth.c	Ensure successful exit status after shutdown
tls/client-tls13-resume.c	Load client cert/key and CA separately
tls/client-tls.c	Load client cert/key and CA separately
staticmemory/memory-bucket-optimizer/tester/Makefile	Add include/lib paths for wolfSSL
staticmemory/memory-bucket-optimizer/optimizer/Makefile	Add include/lib paths for wolfSSL
staticmemory/Makefile	Correct Makefile title/comment
signature/sigtest/Makefile	Make OpenSSL path optional and clearer
SGX_Linux/sgx_t.mk	Update SGX C++ runtime library name
scripts/run-all-examples.sh	Add local manifest-driven runner script
psa/Makefile	Fix include path and guard empty PSA_INCLUDE
pq/stateful_hash_sig/xmss_example.c	Remove redundant include
pq/stateful_hash_sig/lms_example.c	Remove redundant include
pq/ml_kem/README.md	Switch build instructions to `make`
pq/ml_kem/ml_kem.c	Update include/header usage; simplify main signature
pq/ml_kem/Makefile	Add Makefile for ML-KEM example
pq/ml_dsa/ml_dsa.c	Update to FIPS 204 ctx APIs; fix exit code
pq/ml_dsa/Makefile	Fix clean target artifacts list
pkcs7/scripts/runall.sh	Fix ordering + ensure failures exit non-zero
pkcs7/scripts/openssl-verify.sh	Gate on runall, add failure accounting, improve checks
pkcs7/Makefile	Correct Makefile title and zlib comment
pkcs11/Makefile	Correct Makefile title/comment
pk/srp/Makefile	Fix WOLFSSL_INSTALL_DIR prefix
pk/enc-through-sign-rsa/rsa-public-decrypt-app.c	Normalize return code from length-returning API
embedded/Makefile	Adjust linux-specific target list handling
ecc/ecc-params.c	Handle negative lengths; normalize exit status
ebpf/tls-uprobe-trace/Makefile	Add wolfSSL include/lib flags
dtls/server-dtls-rw-threads.c	Fix cert paths for CI execution
dtls/client-dtls-rw-threads.c	Fix cert paths for CI execution
custom-io-callbacks/file-server/file-server.c	Fix server return codes / cleanup flow
custom-io-callbacks/file-server/check.sh	Make server/client orchestration bounded and reliable
custom-io-callbacks/file-client/file-client.c	Fix client return codes
crypto/keywrap/Makefile	Add Makefile for keywrap example
crypto/aes-modes/aes-xts.c	Fix XTS key halves to be distinct
certvfy/Makefile	Add include/lib flags for wolfSSL
certs/crl/regenerate-crl.sh	Add script to regenerate CRL fixture
certs/crl/crl.pem	Update CRL fixture to unexpired version
certmanager/README.md	Fix example invocation name
certmanager/certverify.c	Ensure successful exit status
certmanager/certloadverifybuffer.c	Update embedded cert fixtures
certgen/Makefile	Add Makefile header/comment
certgen/csr_w_ed25519_example.c	Normalize return code from length-returning API
can-bus/Makefile	Add include/lib flags for wolfSSL
can-bus/generate_ssl.sh	Cap CN length to X.509 limits
btle/common/btle-sim.c	Ignore SIGPIPE to avoid FIFO-close crashes
.gitignore	Ignore new binaries/pycache; fix typo
.github/workflows/nightly.yml	Add scheduled nightly orchestration
.github/workflows/examples.yml	Add host-tier matrix workflow + coverage gate
.github/workflows/emulated.yml	Add emulated-tier workflow (TPM/SGX/CAN/UEFI)
.github/workflows/cross-build.yml	Add cross-build workflow (ESP32/Pico/etc.)
.github/workflows/ci-triage.yml	Add triage workflow for nightly results
.github/workflows/_resolve-wolfssl.yml	Add reusable wolfSSL ref resolution workflow
.github/scripts/run_example.py	Add example harness (build/run/assert/output/artifacts)
.github/scripts/manifest.py	Add manifest loader + matrix generator + coverage checker
.github/scripts/issue_store.py	Add issue identity/body/state machine + scrubbing
.github/scripts/ci_triage.py	Add triage logic (retry, classify, issues, AI advisory)
.github/examples-manifest.yml	Add source-of-truth manifest for CI coverage/matrix
.github/actions/setup-wolfssl/action.yml	Add composite action to build/cache/install wolfSSL

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…efore resuming

…on ticket

… the ticket

…t fail CI

…ever masked to 0

…ated hsm.yml job

padelsbach · 2026-07-22T15:39:25Z

+          (key, heap, devId))
+WRAP_VOID(wc_MlDsaKey_Free, (wc_MlDsaKey* key), (key))
+WRAP_FUNC(int, wc_MlDsaKey_SetParams, (wc_MlDsaKey* key, byte level), (key, level))
+WRAP_FUNC(int, wc_MlDsaKey_MakeKey, (wc_MlDsaKey* key, WC_RNG* rng), (key, rng))


the context-less sign and verify are only available under WOLFSSL_MLDSA_NO_CTX according to /git/wolfssl/wolfssl/wolfcrypt/wc_mldsa.h

Thanks! Switched to wc_MlDsaKey_SignCtx/VerifyCtx with an empty context, those are always compiled in.

padelsbach · 2026-07-22T15:42:19Z

AI says these values aren't big enough for RSA 2048, and the check will always fail due to input validation

Bumped to n[256]/d[256]/p[128]/q[128] so the components fit.

padelsbach · 2026-07-22T15:46:59Z

+        wolfssl_ref: ${{ fromJson(needs.resolve.outputs.refs_json) }}
+        include:
+          # uefi-static clones wolfSSL itself; `all` is wolfcrypt.efi
+          - example: uefi-static


uefi-static is not being run. Seems it needs to be one layer up to be part of the matrix:

padelsbach · 2026-07-22T15:52:10Z

+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: results-tpm-${{ matrix.simulator }}-attempt${{ github.run_attempt }}


AI says there should be another ref here, maybe {{ matrix.ref_slug }}?

same with the --results above

Added -${{ matrix.wolfssl_ref }} to the --results file and the artifact name.

padelsbach · 2026-07-22T15:54:31Z

+    steps:
+      - id: pick
+        run: |
+          case "${{ github.event_name }}" in


Apparently, the github.event_name is owned by the caller flow, and would be schedule in this case, which would not trigger this action.

Seems this is still not fixed

Keys off inputs.caller_run_id now, set by nightly.yml:104, not github.event_name.

padelsbach · 2026-07-22T15:59:42Z

+name: CMake
+
+on:
+  push:


Lots of on: push and on: pull_request added. Questions:

Does this trigger 2x runs?

Can we reduce this? At minimum, add draft == false

ya it would run twice if on the same fork I added smoke test on draft and also filters for tests if that examples dir wasn't touched it wont run it on the pr.

I made it push branches: [master] only so a fork PR fires once. Added draft == false and paths filters.

padelsbach · 2026-07-22T16:02:53Z

+    steps:
+      - uses: actions/checkout@v5
+      - uses: ./.github/actions/apt-update
+      - name: Install shellcheck + cppcheck + actionlint


lint.sh does not actually use spellcheck

Added codespell over the repo text.

padelsbach · 2026-07-22T16:11:40Z

+out=$("$WORK/wolfcrypt_test" 2>&1) || true
+kill "$simpid" 2>/dev/null || true
+printf '%s\n' "$out" | tail -40
+printf '%s' "$out" | grep -qiE "Ran wolfCrypt test|Test complete|error test passed" \


AI says this grep will always print success

Now fails on nonzero rc or missing string (se050-run.sh:97), and the demo returns the test result.

padelsbach · 2026-07-22T16:16:18Z

General comment: I'd argue the build+test scripts should not live in the CI dir, and should not be shell scripts. make check for each applicable example would be more canonical and perhaps closer to the end-user environment. (And it's how Provider runs CI as well.)

Ofc, keep the ones that don't fit that model (eg Android)

…n BUFFER_E

padelsbach · 2026-07-22T17:57:30Z

+    steps:
+      - id: pick
+        run: |
+          case "${{ github.event_name }}" in


Seems this is still not fixed

padelsbach · 2026-07-22T17:59:38Z

+makefail=""
+for mk in $(find . -name Makefile -not -path './.git/*' -not -path '*/wolfssl/*' | sort); do
+    d=$(dirname "$mk")
+    make -n -C "$d" >/dev/null 2>&1 || makefail="$makefail $d"


AI sayas this will clone the repo into the tree, and will pollute the lint

padelsbach · 2026-07-22T18:07:03Z

+    path: ecc
+    profile: ecc
+    run:
+      - exec: [./ecc-key-decode]


I think this stuff lives in the example-specific makefiles now, no need to duplicate it here.

In fact, I don't see any make check calls from .github

run_example.py calls make check per dir now (mode: check) and the duplicated exec/expect is gone. 46 dirs...

padelsbach · 2026-07-22T18:09:12Z

+    path: crypto/aes-modes
+    profile: crypto
+    setup:
+      - [sh, -c, "printf 'wolfssl-examples aes-modes CI input\\n' > in.txt"]


I think we need to create the inputs in the makefiles too. Eg in.txt, mem.log.

Moved input creation into the check targets, each one is self-contained now.

padelsbach · 2026-07-22T18:12:51Z

        authEnvelopedDataKEKRI.der compressedData.der \
        smime-created.p7s
+
+check: signedData-stream


I think this check is not sufficient to catch a failure. From the .c:

printf("read %d bytes from file\n", encryptedSz); decryptedSz = signedData_verify(encrypted, encryptedSz, cert, certSz, key, keySz, decrypted, decryptedSz);

Greps 'Successfully verified SignedData bundle' now, which only prints on a passing verify.

padelsbach · 2026-07-22T18:14:21Z

 clean:
 	rm -f $(TARGETS)
+
+check: rsa-nb


I think this also does not check the verify result

Added a 'RSA non-block verify successful' print after the XMEMCMP and grep that.

padelsbach · 2026-07-22T18:17:13Z

+	./ecc-key-decode     | grep -q "Success"
+	./ecc-params         | grep -q "Gy: 32"
+	./ecc-sign           | grep -q "Firmware Signature 9: Ret 0"
+	./ecc-stack          | grep -q "stack used ="


AI says that ecc-stack does not return non-zero on make key failure, and this would pass

padelsbach · 2026-07-22T18:18:59Z

+.PHONY: all clean check
+
+check: memory_bucket_optimizer
+	./memory_bucket_optimizer mem.log | grep -q 'Optimization Summary:'


AI says this will always pass.

Can you review the Makefiles to make sure they check for actual failure?

Every check runs under -eo pipefail now, and fixed four grep/SIGPIPE bugs that hid real failures.

padelsbach · 2026-07-22T18:26:57Z

+    path: tpm
+    tier: emulated
+    profile: all
+    # emulated.yml runs it against wolfTPM's own fwTPM simulator on :2321


AI says this should have a run: field to actually exercise the test. Or should it be build-only?

good finds, fixing these now

… workflow_call

…cks and inputs

…ith ecc

…nputs into the check targets

Copilot AI review requested due to automatic review settings July 15, 2026 00:38

Copilot started reviewing on behalf of aidangarske July 15, 2026 00:38 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

aidangarske force-pushed the ci-matrix branch 5 times, most recently from 56d67ec to f0f3a18 Compare July 15, 2026 16:34

aidangarske mentioned this pull request Jul 15, 2026

Retry the ISO TP flow control wait up to N_Bs wolfSSL/wolfssl#10915

Closed

aidangarske force-pushed the ci-matrix branch from 378a78a to ddcb1c4 Compare July 15, 2026 20:15

aidangarske self-assigned this Jul 16, 2026

aidangarske force-pushed the ci-matrix branch 19 times, most recently from 0ffccd5 to f4fd5e6 Compare July 16, 2026 22:54

aidangarske added 9 commits July 22, 2026 08:53

Accept the zero success return from wolfSSL_CTX_set_max_early_data

997bf02

Read the earlydata reply so the client processes the session ticket b…

5677309

…efore resuming

Read the earlydata reply in the DTLS client so it processes the sessi…

e3610ae

…on ticket

Run the tls13 and dtls13 earlydata pairs now that the clients process…

494bf68

… the ticket

Let expect_fail clear on refs that carry the fix via a fixed_on marker

dcbab04

Retry the PSA TLS 1.3 handshake so an intermittent ECC reject does no…

d63c26b

…t fail CI

Retry network fetches across CI so a transient blip does not fail a job

96f1fad

Normalize do_ecc and do_25519 exit codes like do_448 so an error is n…

2592dbc

…ever masked to 0

Build and run the merged-in hsm dtls_client example in CI via a dedic…

e06674a

…ated hsm.yml job

padelsbach reviewed Jul 22, 2026

View reviewed changes

aidangarske added 10 commits July 22, 2026 09:48

Add a make check target to each applicable example

69db57a

Run only the example and lint smoke set on draft PRs

df4568c

Only run a per-target workflow when its own example dir changes

8c71295

Assert the real se050 wolfcrypt result instead of an early sub-test line

86f07c7

Size the RSA 2048 key export buffers so the UEFI test does not fail o…

a32df9f

…n BUFFER_E

Call the always-present MLDSA context API from the UEFI driver

9eb6617

Cross uefi-static and uefi-library with both wolfSSL refs in the matrix

3f3b0dd

Give each tpm matrix leg a ref-unique results file and artifact

a840e1f

Add a codespell spellcheck pass to the lint job

320e6c9

Run push CI on master only so a PR branch does not double-trigger

0601ffd

padelsbach reviewed Jul 22, 2026

View reviewed changes

aidangarske added 8 commits July 22, 2026 12:19

Select valgrind by caller_run_id since event_name is the caller under…

5ea19d3

… workflow_call

Make example check targets catch real failures with pipefail exit che…

5edb6f1

…cks and inputs

Wire the harness to run make check for mode check examples starting w…

2024add

…ith ecc

Migrate the single-entry exec examples to mode check and fold their i…

4871c14

…nputs into the check targets

Fix four make check assertions that misfired under pipefail

818d1cd

Give the tpm manifest entry a run step so it asserts output

c11774d

Skip uefi-static in the lint make -n loop so it does not clone

a629a78

Assert the actual verify result in the pkcs7 and rsa-nb checks

8b20297

Uh oh!

Conversation

aidangarske commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI

Fixes

Upstream wolfSSL bugs this CI surfaced

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

padelsbach commented Jul 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aidangarske commented Jul 15, 2026 •

edited

Loading