Silent Payments - Full Implementation#134
Open
odudex wants to merge 61 commits into
Open
Conversation
Reject non-int and reserved/out-of-range labels (m=0 is change), drop the non-standard str/bytes label support, correct swapped scan/spend keys in the network test, and apply black formatting.
- Subclass EmbitError so existing except EmbitError callers still catch malformed-address errors (e.g. script.address_to_scriptpubkey). - Restore the BIP-141 2-40 byte witness program limit in decode() and drop the sp/tsp special-casing; SP addresses are not witness programs. - Replace the 118 length cap with BIP-352's recommended 1023 so valid higher-version silent payment addresses decode; cite BIP-173/BIP-352. - Drop the no-placeholder f-string and the encode() try/except that masked the original Bech32DecodeError.
Drop the typing and collections imports (Counter/defaultdict, Tuple/List/Dict) and convert COutPoint from typing.NamedTuple to a plain class with positional int.to_bytes. Replace remaining f-strings with str.format and remove the now unused COutPoint import from bip352.
- Remove bech32_decode_long; the flexible bech32_decode (this branch) already
handles long strings, so decode SP keys and silent payment addresses through
it, converting Bech32DecodeError to the caller's error type.
- Drop the now-dead 'is None' guards (convertbits raises instead of returning
None) and wrap conversion to keep DescriptorError/ValueError consistent.
- Seek to an absolute offset after 'sp(' so parsing is robust to a short read.
- Raise a clear DescriptorError for unsupported address/derive/script_pubkey on
SilentPaymentDescriptor, and cover the Descriptor.from_string('sp(...)') path.
The new pure-Python secp256k1 tweak functions require a bytearray (they mutate in place via index assignment); ec_pubkey_parse returns bytes, which only worked with the ctypes backend. Wrap parsed points in bytearray so create_outputs works on both backends (fixes the pure-Python CI path).
…rsing - Add LOCKTIME_THRESHOLD constant (500000000) for locktime type discrimination - Add required_time_locktime and required_height_locktime fields to InputScope - Propagate those fields in InputScope.update() - Parse PSBT_IN_REQUIRED_TIME_LOCKTIME (0x11) and PSBT_IN_REQUIRED_HEIGHT_LOCKTIME (0x12) in InputScope.read_value(), with range validation - Serialize 0x11 and 0x12 in InputScope.write_to() under the version==2 block - Add version=None parameter to InputScope.read_value() and OutputScope.read_value() so callers can signal the PSBT version being parsed - Add InputScope.read_from() and OutputScope.read_from() classmethods that accept and forward the version parameter (supersede PSBTScope base classmethod for v2) - Propagate version in LInputScope.read_value() and LOutputScope.read_value() (pset.py) so the Liquid subclasses stay compatible - Pass version=self.version when PSBTView calls read_from() for input/output scopes so PSBTv2 scopes are parsed with the correct version context
- Add TxModifiable class with INPUTS/OUTPUTS/SIGHASH_SINGLE bit flags - Add PSBT._validate_v2_output() classmethod to check required PSBTv2 output fields; PSET overrides it to allow value_commitment in place of value - Add tx_modifiable_flags field to PSBT.__init__() (None for v0/unset) - Update comment on self.version to note '2 for v2' - Add PSBT.determine_locktime(): implements BIP-370 locktime selection algorithm (height vs time preference, max-of-minimums) - Update PSBT.tx property to call determine_locktime() for PSBTv2 - Update PSBT.write_to(): serialize PSBT_GLOBAL_TX_MODIFIABLE (0x06) when set - Rewrite PSBT.read_from(): collect all global KVs into OrderedDict before branching on PSBTv2 vs PSBTv0; detect version from 0xfb; reject unsigned tx in PSBTv2 and reject v2-only keys in PSBTv0; validate required fields after parsing; call read_from with version=version on all scopes - Rewrite PSBT.parse_unknowns(): strip 0xfb/0x06; guard 0x02/0x03/0x04/0x05 behind version==2; track _raw_input_count_from_global and _raw_output_count_from_global for PSBTv2 scope parsing - Add version guards to InputScope.read_value() for 0x0e/0x0f/0x10: PSBT_IN_PREVIOUS_TXID, PSBT_IN_OUTPUT_INDEX, and PSBT_IN_SEQUENCE are rejected when version != 2 - Add version guards to OutputScope.read_value() for 0x03/0x04: PSBT_OUT_AMOUNT and PSBT_OUT_SCRIPT are rejected when version != 2
…_with - Update sign_with(): after each legacy/segwit signature, update tx_modifiable_flags per BIP-370: clear INPUTS bit if not ANYONECANPAY, clear OUTPUTS bit if not NONE, set SIGHASH_SINGLE bit if sighash type is SINGLE - Add get_tx_modifiable() / set_tx_modifiable(flags): getter/setter for PSBT_GLOBAL_TX_MODIFIABLE; setter raises PSBTError on PSBTv0 - Add is_inputs_modifiable(): True if version!=2, tx_modifiable_flags is None, or INPUTS bit is set - Add is_outputs_modifiable(): True if version!=2, tx_modifiable_flags is None, or OUTPUTS bit is set - Add has_sighash_single(): True only if version==2, tx_modifiable_flags is not None, and SIGHASH_SINGLE bit is set - Add add_input(input_scope): appends an InputScope, gated on is_inputs_modifiable(); updates _raw_input_count_from_global for PSBTv2 - Add add_output(output_scope): appends an OutputScope, gated on is_outputs_modifiable(); updates _raw_output_count_from_global for PSBTv2
Add test_psbtV2.py with 48 tests covering PSBTv2 parsing, serialization, locktime selection, TxModifiable flags, and add_input/add_output API.
- reject BIP370-only globals in PSBTView v0 parsing - preserve signed and zero tx versions when synthesizing transactions - serialize transaction versions as signed int32 - keep PSBTv2 compressed non-witness UTXO parsing key-order independent - add regression coverage for the follow-up findings
Replace new PSBTv2/PSET f-strings with old-style formatting so the branch remains parseable on older MicroPython versions.
Replace signed=True usage in BIP370 signed integer parsing and serialization with helper functions compatible with older MicroPython byte conversion APIs. This keeps PSBTv2 transaction versions and output amounts spec-compliant without changing regular keyword-argument call sites.
GlobalTransactionView.version read the version unsigned while the sighash paths re-encode it via _signed_to_bytes, raising OverflowError for v0 txs whose version has the high bit set. Match Transaction.read_from.
The count-based guard in add_input rejected legitimate end-appends. BIP370 only requires preserving input/output pairing for SIGHASH_SINGLE-signed inputs; appending never reindexes existing pairs, so it is always safe. Now symmetric with add_output.
write_to merged extra_input/output_streams with default v0 parsing, which would misclassify the version-sensitive PSBTv2 fields (0x0e-0x12, 0x03-0x04). Parse them with the view's own version.
The 'or 0xFFFFFFFF' fallback rewrote a stored sequence of 0 (valid, enables nLockTime) to 0xFFFFFFFF, altering the assembled transaction and its sighashes. PSBTv2 parses PSBT_IN_SEQUENCE explicitly, making this reachable.
InputScope.update and OutputScope.update overwrote taproot_internal_key unconditionally, wiping an existing key when combining with a scope that lacks one. Guard like the neighboring fields.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR introduces full Silent Payments support across embit, including BIP-352 address/output derivation, BIP-374 DLEQ proofs, BIP-375 PSBT fields + validation, BIP-376 spend-from flow, and BIP-392 sp() descriptors. It also extends the secp256k1 fallback backends and PSBTv2 handling to support the new SP/PSBT requirements, with extensive new test coverage and official vector fixtures.
Changes:
- Add Silent Payments core modules (BIP-352/374/375/376) and PSBTv2 integration (SP-aware scopes, signing hook, validator).
- Extend descriptor parsing with BIP-392
sp()descriptors and SP key encodings. - Improve bech32 error handling + length limits, secp256k1 fallback functionality, and add comprehensive tests/vectors for SP + DLEQ + PSBTv2.
Reviewed changes
Copilot reviewed 33 out of 36 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/tests/test_sp_descriptor.py | Adds tests for BIP-392 sp() descriptors and SP key encoding/roundtrips. |
| tests/tests/test_psbt_signer_sp.py | Adds end-to-end signer flow tests for BIP-375/376 SP PSBT signing (incl. taproot + aux_rand). |
| tests/tests/test_ecc.py | Adds coverage for PrivateKey.even_y() normalization behavior. |
| tests/tests/test_dleq.py | Runs official BIP-374 DLEQ vectors (including pure-Python secp backend forcing). |
| tests/tests/test_bip375_vectors.py | Validates official BIP-375 PSBT vectors through the validator. |
| tests/tests/test_bip352.py | Adds BIP-352 address/label/output derivation tests and send/receive vectors. |
| tests/tests/test_bindings.py | Adds/extends tests for secp256k1 tweak-mul and pubkey combine fallbacks + parity checks. |
| tests/tests/test_bech32.py | Updates bech32 tests to expect exceptions; adds SP bech32m decoding test and round-trips. |
| tests/tests/data/test_vectors_verify_proof.csv | Adds BIP-374 verify-proof vector fixtures. |
| tests/tests/data/test_vectors_generate_proof.csv | Adds BIP-374 generate-proof vector fixtures. |
| src/embit/util/py_secp256k1.py | Implements missing tweak-mul and pubkey-combine fallback operations; adds mutability enforcement. |
| src/embit/util/key.py | Makes ECPubKey.set() return self to support fluent usage patterns. |
| src/embit/util/ctypes_secp256k1.py | Improves bytearray compatibility for ctypes bindings via readable/writable buffer helpers. |
| src/embit/transaction.py | Adds signed int32 tx-version helpers and introduces COutPoint for SP input hashing. |
| src/embit/silent_payments/validator.py | Introduces BIP-375 PSBT validation pipeline (structure/ECDH/eligibility/output scripts). |
| src/embit/silent_payments/psbt.py | Adds SP-aware PSBT scopes + SilentPaymentsPSBT with SP signing and BIP-376 finalizer. |
| src/embit/silent_payments/fields.py | Defines shared SP field/validation errors and SilentPaymentData. |
| src/embit/silent_payments/ecdh.py | Implements SP ECDH share computation, DLEQ proof generation, and eligible-input logic. |
| src/embit/silent_payments/dleq.py | Implements BIP-374 DLEQ proof generation/verification. |
| src/embit/silent_payments/bip352.py | Implements BIP-352 address encoding/decoding and output derivation helpers. |
| src/embit/silent_payments/init.py | Exposes Silent Payments public API surface. |
| src/embit/psbtview.py | Enhances PSBT view streaming for PSBTv2 globals, locktime determination, and tx_modifiable updates. |
| src/embit/psbt.py | Major PSBTv2 support expansion (globals parsing, locktime rules, tx_modifiable, signed fields handling). |
| src/embit/liquid/transaction.py | Fixes tweak-mul callsite to use mutable pubkey buffers. |
| src/embit/liquid/pset.py | Threads PSBT version through read_value; fixes tweak-mul callsites; adjusts v2 output validation for blinded outputs. |
| src/embit/liquid/descriptor.py | Adjusts MuSig combine helpers to use mutable buffers compatible with new tweak APIs. |
| src/embit/ec.py | Adds BIP-352 spend tweaking and even_y() key normalization. |
| src/embit/descriptor/sp.py | Adds SP key encoding/decoding + SilentPaymentDescriptor implementation (BIP-392). |
| src/embit/descriptor/descriptor.py | Adds dispatch for sp(...) descriptors in generic descriptor parsing. |
| src/embit/descriptor/init.py | Exports SilentPaymentDescriptor. |
| src/embit/bech32.py | Switches bech32/segwit address functions to raise structured exceptions and expands max length. |
| .gitignore | Ignores Claude-related files/dir. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Silent payment (sp/tsp) addresses are not witness programs and must not be | ||
| decoded here; use bech32_decode + convertbits for those. | ||
| """ | ||
| encoding, hrpgot, data = bech32_decode(addr) |
| raise PSBTError("PSBT_OUT_AMOUNT must be 8 bytes") | ||
| if self.value is not None: | ||
| raise PSBTError("Duplicated PSBT_OUT_AMOUNT") | ||
| self.value = _signed_from_bytes(v) |
Comment on lines
+510
to
+513
| try: | ||
| eligible = get_eligible_inputs(self.inputs, has_sp_outputs=True) | ||
| except SPValidationError: | ||
| return 0 |
| return bech32.bech32_encode(bech32.Encoding.BECH32M, hrp, [version] + data) | ||
|
|
||
|
|
||
| # TODO: use the bech32 decode function once the flexible bech32 PR is in |
Comment on lines
+67
to
+72
| if address.startswith("sp1"): | ||
| hrp = "sp" | ||
| elif address.startswith("tsp1"): | ||
| hrp = "tsp" | ||
| else: | ||
| raise ValueError("Invalid silent payment address: unknown HRP") |
Comment on lines
+146
to
+150
| with pytest.raises(Exception): | ||
| for label in INVALID_LABEL_TEST_VECTORS: | ||
| bip352.generate_silent_payment_address( | ||
| scan_priv_key, spend_priv_key.get_public_key(), label | ||
| ) |
Collaborator
Author
|
@copilot ensure the issues you pointed were addressed. |
All 6 issues have been addressed in commits
All 15 relevant tests pass. |
- _validate_ecdh_coverage now raises SPValidationError when a scan key carries a PSBT_GLOBAL_SP_DLEQ proof but no matching PSBT_GLOBAL_SP_ECDH_SHARE. Previously this hit an unguarded dict index in _verify_global_dleq_proof and raised a bare KeyError, breaking the documented "raises SPValidationError" contract. Add a regression test derived from the official global-share vectors. - minor fixes on code readability Signed-off-by: notTanveer <tanveer.x.ansari@gmail.com>
Comment on lines
+108
to
+112
| def get_input_hash(outpoints, sum_pubkey_bytes: bytes) -> bytes: | ||
| lowest_outpoint = sorted(outpoints, key=lambda o: o.serialize())[0] | ||
| preimage = lowest_outpoint.serialize() + sum_pubkey_bytes | ||
| return tagged_hash("BIP0352/Inputs", preimage) | ||
|
|
Comment on lines
+235
to
+236
| def test_empty_sp(self): | ||
| self.assertRaises(Exception, SilentPaymentDescriptor.from_string, "sp()") |
get_input_hash() now rejects an empty outpoints list with a ValueError instead of crashing with IndexError. _read_sp_key_expression() rejects an empty key expression with DescriptorError instead of an opaque WIF checksum ValueError, and test_empty_sp asserts DescriptorError.
…_shares flag - create_outputs: iterate recipient occurrences in list order instead of collapsing to counts, so k is assigned correctly when addresses sharing a scan key are interleaved (matches BIP-375 validator's derivation order) - sign_with: add with_sp_shares=True param to allow callers to skip SP ECDH share generation independently of SP spend signing
…t ordering - create_outputs: add regression test asserting k follows recipients (vout) order when addresses share a scan key - create_outputs: document the FIFO return-mapping contract in the docstring - derive_silent_payment_outputs: strip trailing whitespace
Comment on lines
+155
to
+165
| # Rewind to before the token via a relative seek (no tell(); MicroPython's | ||
| # BytesIO lacks it). Only `token` was consumed since; the delimiter, if any, | ||
| # was already pushed back above. | ||
| s.seek(-len(token), 1) | ||
| if origin: | ||
| origin_str = "[%s]" % origin | ||
| combined = BytesIO(origin_str.encode() + s.read()) | ||
| key = Key.read_from(combined) | ||
| else: | ||
| key = Key.read_from(s) | ||
| return key, None |
Comment on lines
+270
to
276
| if key == b"\x00": | ||
| # we found global transaction; defer version==2 check until after the loop | ||
| # so that PSBT_GLOBAL_UNSIGNED_TX is rejected even when it appears before | ||
| # PSBT_GLOBAL_VERSION (key order is not guaranteed). | ||
| if (num_inputs is not None) or (num_outputs is not None): | ||
| raise PSBTError("Invalid global transaction") | ||
| tx_len = compact.read_from(stream) |
The standard-Key branch built a temporary BytesIO from s.read(), draining the original stream to EOF and breaking parsing of the comma/closing paren in two-arg sp([origin]scan, spend). Rewind past the origin prefix and let Key.read_from(s) consume the expression in-place instead.
A second PSBT_GLOBAL_UNSIGNED_TX (key 0x00) silently overwrote tx_offset/num_inputs/num_outputs. Guard on tx_offset being set and raise instead, matching the adjacent Duplicate global key check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains a full Silent Payment implementation. It replaces PR# 131(partial SP), covers PRs #64, #87, #95, #96, #125, #126, #130, #133 with several adjusts and fixes on top of them.