Summary
When a guardian-co-signed account performs concurrent / rapidly-overlapping transactions (e.g. a consume_notes while the same account is also being mutated by another in-flight tx), canonicalization of the pending delta stalls for minutes — up to the full submission_grace_period_seconds (600s in prod) — and the account is locked the whole time, returning 409 ConflictPendingDelta to every new proposal. Serializing operations per account makes the problem disappear entirely.
This was found while integrating the guardian into the Miden Wallet (hosted guardian.openzeppelin.com, Miden 0.15 testnet, @openzeppelin/guardian-client@0.15.0-rc.0). Code references below are against guardian@89591a3 (current main).
Measurements (these rule out the obvious suspects)
- Testnet block production: ~2.9 s/block (fast).
- Guardian HTTP latency:
/pubkey ~0.2 s; every /state, /delta/proposal, /delta request ~125–170 ms (fast).
- Yet consume finalization (time from
POST /delta/proposal until the co-signed delta is canonical / visible via /state) under a concurrent workload: p50 ~5.5 s but p90 = 216 s, max = 586 s, and 13% of POST /delta/proposal returned 409.
So it's neither network latency nor block time — it's canonicalization waiting on a commitment that never matches.
Root cause (as I read the source)
POST /delta makes the delta a Candidate and sets has_pending_candidate = true. Any subsequent POST /delta/proposal for that account is rejected with 409 ConflictPendingDelta — crates/server/src/services/push_delta_proposal.rs:79 (has_pending_candidate) → :91 (return Err(GuardianError::ConflictPendingDelta)).- The canonicalization worker (every
check_interval_seconds = 10) calls verify_state → get_account_commitment() RPC and finalizes only when the on-chain commitment equals the guardian's locally-computed expected commitment — crates/server/src/network/miden/mod.rs:108, crates/server/src/jobs/canonicalization/processor.rs:151-171.
- On a mismatch within the grace period, it just defers without consuming retry budget —
processor.rs:174-189 (candidate_age_seconds < self.submission_grace_period_seconds → "Delta verification failed during submission grace period; will retry without consuming retry budget"). Prod config: CanonicalizationConfig::new(10, 48).with_submission_grace_period_seconds(600) — crates/server/src/main.rs:40.
The trap: the expected commitment is computed for base_state + this_delta. If the account legitimately advances on-chain via another tx before this candidate is verified, the actual commitment is base_state + other_delta(s) + this_delta (or a different ordering) and never equals the guardian's expectation — so the candidate burns the entire 600 s grace and is then discarded, blocking the account (409) the whole time.
Controlled experiment (concurrency is the trigger)
Identical network + hosted guardian; only the wallet's operation pattern changed:
| metric |
concurrent workload |
serialized (one op per account at a time) |
409 ConflictPendingDelta |
16 / 127 (13%) |
0 / 32 (0%) |
/delta/proposal finalize gap p90 / max |
216 s / 586 s |
43.5 s / 57.4 s |
/state polls (client waiting) |
1,879 |
298 |
| result |
crawled, never converged |
passed, balances conserved, settled in 4 s |
Suggested direction
When verify_state fails because the account has advanced past the expected commitment (vs. a genuine verification failure), canonicalization could re-derive the expected commitment from the current on-chain account state and re-check, rather than waiting out the grace period against a now-stale expectation. At minimum, distinguishing "account moved on / superseded" from "invalid" would let the server release has_pending_candidate quickly instead of holding the account locked for up to 600 s. Happy to provide the full request/timeline traces.
Environment
- guardian: hosted
guardian.openzeppelin.com; source main@89591a3
- client:
@openzeppelin/guardian-client@0.15.0-rc.0, @openzeppelin/miden-multisig-client
- chain: Miden 0.15 testnet (
rpc.testnet.miden.io, blocks ~2.9 s)
Summary
When a guardian-co-signed account performs concurrent / rapidly-overlapping transactions (e.g. a
consume_noteswhile the same account is also being mutated by another in-flight tx), canonicalization of the pending delta stalls for minutes — up to the fullsubmission_grace_period_seconds(600s in prod) — and the account is locked the whole time, returning409 ConflictPendingDeltato every new proposal. Serializing operations per account makes the problem disappear entirely.This was found while integrating the guardian into the Miden Wallet (hosted
guardian.openzeppelin.com, Miden 0.15 testnet,@openzeppelin/guardian-client@0.15.0-rc.0). Code references below are againstguardian@89591a3(currentmain).Measurements (these rule out the obvious suspects)
/pubkey~0.2 s; every/state,/delta/proposal,/deltarequest ~125–170 ms (fast).POST /delta/proposaluntil the co-signed delta is canonical / visible via/state) under a concurrent workload: p50 ~5.5 s but p90 = 216 s, max = 586 s, and 13% ofPOST /delta/proposalreturned409.So it's neither network latency nor block time — it's canonicalization waiting on a commitment that never matches.
Root cause (as I read the source)
POST /deltamakes the delta a Candidate and setshas_pending_candidate = true. Any subsequentPOST /delta/proposalfor that account is rejected with409 ConflictPendingDelta—crates/server/src/services/push_delta_proposal.rs:79(has_pending_candidate) →:91(return Err(GuardianError::ConflictPendingDelta)).check_interval_seconds = 10) callsverify_state→get_account_commitment()RPC and finalizes only when the on-chain commitment equals the guardian's locally-computed expected commitment —crates/server/src/network/miden/mod.rs:108,crates/server/src/jobs/canonicalization/processor.rs:151-171.processor.rs:174-189(candidate_age_seconds < self.submission_grace_period_seconds→ "Delta verification failed during submission grace period; will retry without consuming retry budget"). Prod config:CanonicalizationConfig::new(10, 48).with_submission_grace_period_seconds(600)—crates/server/src/main.rs:40.The trap: the expected commitment is computed for base_state + this_delta. If the account legitimately advances on-chain via another tx before this candidate is verified, the actual commitment is
base_state + other_delta(s) + this_delta(or a different ordering) and never equals the guardian's expectation — so the candidate burns the entire 600 s grace and is then discarded, blocking the account (409) the whole time.Controlled experiment (concurrency is the trigger)
Identical network + hosted guardian; only the wallet's operation pattern changed:
409 ConflictPendingDelta/delta/proposalfinalize gap p90 / max/statepolls (client waiting)Suggested direction
When
verify_statefails because the account has advanced past the expected commitment (vs. a genuine verification failure), canonicalization could re-derive the expected commitment from the current on-chain account state and re-check, rather than waiting out the grace period against a now-stale expectation. At minimum, distinguishing "account moved on / superseded" from "invalid" would let the server releasehas_pending_candidatequickly instead of holding the account locked for up to 600 s. Happy to provide the full request/timeline traces.Environment
guardian.openzeppelin.com; sourcemain@89591a3@openzeppelin/guardian-client@0.15.0-rc.0,@openzeppelin/miden-multisig-clientrpc.testnet.miden.io, blocks ~2.9 s)