Replace bespoke JSON tx-spec with CCL TxPlan (YAML): offline build, Plutus, full intent coverage#2
Merged
Merged
Conversation
Upgrade CCL 0.7.2 -> 0.8.0-pre4 (backward compatible; only the deprecated
ScriptTx path changes) and adopt CCL's native TxPlan YAML format.
- Delete the bespoke spec + mappers + provider path (~2400 LOC):
TxSpec, TxOperation, TxItemSpec, TxSpecMapper, ScriptTxSpecMapper,
ProviderConfig, Yaci{Utxo,ProtocolParams}Supplier, YaciTransactionEvaluator.
- Rewrite QuickTxService to: TxPlan.from(yaml) -> QuickTxBuilder(static
utxoSupplier, () -> protocolParams, null) -> compose(plan) -> build()
-> CBOR. Fully offline, never submits. Reuses StaticUtxoSupplier.
- New entrypoint signature: ccl_quicktx_build(thread, yaml, utxos_json,
protocol_params_json); result stays {tx_cbor, tx_hash, fee} JSON.
- Add --initialize-at-build-time=org.yaml.snakeyaml for native-image.
- Rewrite QuickTxApiTest to build TxPlan YAML txs offline (payments,
multi-intent, variable substitution, insufficient-funds).
Plutus script txs are deferred (no offline exec-unit evaluator in pre4).
Wrappers still target the old signature and are updated next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
End-to-end proven: TxPlan YAML -> offline build -> CBOR, through the Go
wrapper and native lib.
- Go: delete the fluent builder (~1640 LOC); QuickTxApi.Build(yaml, utxos,
protocolParams) marshals the chain data to JSON and calls the new
3-arg ccl_quicktx_build. Example rewritten to TxPlan YAML.
- native-image: add reflect-config for CCL's TxPlan deserialization
classes (TransactionDocument + nested, all 27 intent classes, Amount).
Jackson cannot construct these reflectively in a native image otherwise
("Cannot construct instance of TransactionDocument").
- QuickTxApi now surfaces the wrapped root cause in error messages
(the generic "Failed to deserialize YAML" hid the real problem).
Verified: `go run ./examples/transaction` builds + signs a payment from
YAML offline. Go tests + the other wrappers are updated next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ccl_test.go: replace the builder-based QuickTx unit tests with TxPlan YAML builds (simple/multi payment, variable substitution, insufficient funds); keep the reusable testProtocolParams/makeUtxos/assertTxResult helpers and all non-QuickTx tests. - quicktx_integration_test.go: rewrite the DevKit build->sign->submit tests to YAML; drop the provider-config tests (provider is deferred). - README: document bridge.QuickTx.Build(yaml, utxos, protocolParams); remove the deleted Amount helpers. go vet clean; go test green (unit builds offline, integration skips without DevKit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per decision, ccl_quicktx_build now returns {tx_cbor, tx_hash, fee} as a
YAML document (via CCL's YamlSerializer), matching the YAML input.
- core: QuickTxService serializes the result with YamlSerializer;
QuickTxApiTest parses it with the YAML mapper.
- Go: TxResult uses yaml tags; Build parses the result with gopkg.in/yaml.v3
(aliased goyaml to avoid clashing with the `yaml` parameter). Adds the
yaml.v3 dependency.
Verified: core tests green; `go run ./examples/transaction` round-trips
YAML in -> YAML out (build + sign); go test green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same pattern as Go. Delete the ~1700-line fluent builder (quicktx.py) and the provider module; QuickTx.build(yaml, utxos, protocol_params) calls the 3-arg ccl_quicktx_build and parses the YAML result via pyyaml. - _ffi.py: ccl_quicktx_build argtypes -> 3 char* (yaml, utxos, pp). - __init__.py: export only QuickTx (drop builder/provider classes). - pyproject.toml: add pyyaml dependency. - tests: test_quicktx.py + test_quicktx_integration.py rewritten to YAML; delete provider/compose/new-features integration tests (builder/provider based — YAML equivalents are a follow-up; the core governance fixes stay in the Java). - example + README updated to the YAML API. Verified: 47 passed / 8 skipped (integration skips without DevKit); example builds + signs a payment from TxPlan YAML offline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same pattern as Go/Python. Delete the ~2140-line fluent builder from lib.rs (TxBuilder/ScriptTxBuilder/ComposeTxBuilder/Amount/etc.); QuickTxApi::build(yaml, utxos, protocol_params) calls the 3-arg ccl_quicktx_build and parses the YAML result via serde_yaml. - ffi.rs: ccl_quicktx_build -> 3 char* args. - Cargo.toml: add serde_yaml; drop now-unused imports. - tests: integration_test.rs + quicktx_integration_test.rs QuickTx tests rewritten to YAML (offline unit + DevKit), provider/metadata dropped. - example + README updated to the YAML API. Verified: cargo test green (29 offline tests incl. YAML builds; DevKit tests skip); `cargo run --example transaction` round-trips YAML in -> out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Last wrapper. Delete the ~1300-line fluent builder from index.js (TxBuilder/ScriptTxBuilder/ComposeTxBuilder/Amount/etc.) and the provider module; QuickTxApi.build(yaml, utxos, protocolParams) calls the 3-arg ccl_quicktx_build and parses the YAML result via the `yaml` package. - index.js: ccl_quicktx_build FFI -> 3 cstrings; drop provider export. - package.json: add `yaml` dependency. - index.d.ts: replace builder/provider types with the thin QuickTxApi. - tests: ccl.test.js QuickTx tests + quicktx.integration.test.js rewritten to YAML; delete provider/compose/new-features integration tests. - example + README updated to the YAML API. Verified: bun test green (47 pass / 0 fail; integration skips without DevKit); `bun examples/transaction.js` round-trips YAML in -> out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRs into develop (e.g. the TxPlan refactor) got no CI because the workflows only triggered on main. Add develop to both the unit CI and the DevKit integration-tests triggers so feature->develop PRs run the full matrix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The doc described the deleted bespoke JSON spec. Rewrite it for the TxPlan YAML flow: the new ccl_quicktx_build(yaml, utxos, protocol_params) signature, the YAML result, the TxPlan document structure, the real intent `type` discriminators, verified payment/variable examples, the caller-supplied UTXO/protocol-params JSON, and per-wrapper build+sign snippets. Notes provider removal and the deferred Plutus path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first full PR run went red: Python tests failed with ModuleNotFoundError: yaml, and the JS tests with Cannot find package 'yaml' — CI never installed the YAML parsers the wrappers gained in this refactor. - ci.yml + integration-tests.yml: pip install pyyaml alongside pytest. - wrappers/js/build.gradle: `bun install` before `bun test` (unit + integration) so the `yaml` package is present. Go (yaml.v3) and Rust (serde_yaml) are fetched automatically by go test / cargo test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The metadata intent's value is a scalar string the deserializer
auto-detects; passing it as a JSON string ('{"674": {...}}') is the
working shape (the earlier nested-map attempt threw a YAML parse error).
- QuickTxApiTest: add paymentWithMetadata — builds a payment+metadata
TxPlan and asserts the tx body carries an auxiliary data hash, i.e.
the metadata is actually attached (not just parsed). Confirmed in the
native lib too (reflect-config already covers the metadata classes).
- docs/quicktx.md: add the verified metadata YAML example.
Closes the deferred metadata follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plutus spends/mints need each redeemer's execution units (mem + CPU steps), which requires running the script in a UPLC evaluator. Rather than embed an evaluator, the bridge takes the units as a fourth caller-supplied input — exactly like UTXOs and protocol parameters — and wires CCL's StaticTransactionEvaluator to stamp them onto the redeemers, fully offline. The caller computes them with whatever they like (Ogmios, Blockfrost, Aiken, Scalus). See TODO §2b for the planned pick-and-choose evaluator helpers/examples. - core: QuickTxService.buildTransaction gains exec_units_json -> List<ExUnits> -> withTxEvaluator(StaticTransactionEvaluator); ccl_quicktx_build entrypoint is now 4-arg (yaml, utxos, protocol_params, exec_units). - native-image: register the Plutus reflection that Jackson needs — RedeemerTag / PlutusVersion enums (their @JsonProperty string forms), the plutus.spec types, and the plutus.spec.serializers (custom PlutusData ser/deser). Without these a script build fails in the native image though it passes on the JVM. - wrappers: optional 4th arg through all four — Python/JS default param, Go variadic, Rust Option<&Value>. Existing 3-arg calls unchanged (Go/JS/Python); Rust call sites pass None. - tests: QuickTxApiTest builds a real Plutus mint (always-succeeds V2 policy) offline and asserts the redeemer carries the supplied units, and that it fails without them; Python wrapper has the same pair. Verified end-to-end in the native lib (build OK with units, -10 without). - docs/quicktx.md + TODO.md updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restores the coverage dropped in the builder->TxPlan migration, where it matters most: the native + wrapper path (the JVM/TxPlan layer is CCL's own, covered upstream). - QuickTxIntentsTest (JVM): builds each op with CCL, serializes via TxPlan.from(tx).toYaml(), builds it through the bridge, and emits the exact YAML to build/intent-yamls/<name>.yaml as a fixture. Covers stake_registration/deregistration/delegation/withdrawal, donation, drep_registration/deregistration/update, voting, voting_delegation, governance_proposal (11). - wrappers/go/ccl/intents_test.go: table-driven test that drives every testdata/intents/*.yaml fixture through the native library via the Go wrapper, asserting tx_cbor/tx_hash/fee. This is the real bridge check — it confirms the native-image reflection config covers the governance/ staking/DRep classes (it does; these intents serialize to string fields). 11/11 pass end-to-end in the native lib. Pools + Plutus spend next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pool_registration / pool_update / pool_retirement added to the intent fixtures and Go E2E. Pool registration surfaced native-image reflection gaps the JVM test cannot — exactly what the Go E2E is for: - util.serializers (HexToByteArrayDeserializer / ByteArrayToHexSerializer / InetAddress*) for the byte-array + relay-IP fields, and - transaction.spec.cert.* + spec.UnitInterval for the certificate / margin types. All 14 intent fixtures now build end-to-end through the native lib via Go. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Plutus spend collects from a script-address UTXO with a redeemer + datum and attaches the spending validator. QuickTxIntentsTest builds it and emits the fixture; script_spend_test.go drives it through the native lib with the script UTXO (+ datum hash), a fee/collateral UTXO, and the caller-supplied execution units — asserting it builds with units and fails without. Reuses the Plutus reflection config from the mint (no new native config needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the last intent types to the fixtures + Go end-to-end table: native `minting` (NativeScript policy), `native_script` attachment, `collect_from` (explicit input selection), and `reference_input` (read-only inputs). The Go table now supplies a second small UTXO for the reference-input fixture to read. Every TxPlan intent type is now exercised end-to-end through the native library via Go (18 in the table + Plutus spend + the payment/metadata/ mint cases elsewhere). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds script_minting to the fixtures + a Go E2E test (build with exec units, fail without), mirroring the spend. Go now exercises every TxPlan intent type end-to-end through the native library. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a metadata fixture (payment + attached CIP-20 metadata) to the Go E2E table. Every TxPlan intent type is now exercised end-to-end through the native library via Go. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tures
Brings the per-intent native end-to-end coverage that Go had to every
wrapper, so each language proves it builds all TxPlan intents through the
native library.
- test-fixtures/quicktx-intents/: shared, version-controlled fixtures
(19 generic intents + plutus/{script_minting,script_collect_from}),
generated by the JVM QuickTxIntentsTest. Single source of truth.
- Go: repointed intents_test.go / script_spend_test.go at the shared dir;
dropped the duplicated wrappers/go/ccl/testdata copies.
- Python (test_quicktx_intents.py), Rust (intents_test.rs), JS
(intents.e2e.test.js): table-driven tests over the shared fixtures +
dedicated Plutus mint/spend tests (build with exec units, fail without).
- JS gradle test task now also runs intents.e2e.test.js.
All four wrappers: every intent type builds end-to-end. No native changes
needed — the reflection config proven by the Go suite already covers it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bridge has been on 0.8.0-pre4 since the TxPlan refactor, but README, CLAUDE.md, and TODO.md still said 0.7.2. Mark the 0.7.2->0.8.0 upgrade item done and re-frame TODO §6 (the upstream modules are all available on the current dependency now). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Compose was a feature of the old bespoke format that the TxPlan migration left untested. TxPlan's `transaction` list supports it natively (multiple `tx` entries, each with its own `from`, one fee_payer). - QuickTxIntentsTest.compose builds a 2-sender compose via TxPlan.from(List<AbstractTx>).toYaml() and emits the compose.yaml fixture. - All four wrapper intent tables pick it up; each supplies a second sender's UTXO so the compose builds end-to-end through the native lib. - docs/quicktx.md gets a compose example. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Governance/staking intents build offline but couldn't be submitted: their certificates must be witnessed by the stake (or DRep) key, while ccl_account_sign_tx adds only the payment key — the node rejects them with MissingVKeyWitnessesUTXOW. - core: new ccl_account_sign_tx_multi(..., keys) signs with any subset of payment/stake/drep/committee_cold/committee_hot (CCL Account.signWith*Key), applied in order. The original ccl_account_sign_tx is unchanged. - wrappers: sign_tx_with_keys (Python/Rust), SignTxWithKeys (Go, variadic roles), signTxWithKeys (JS); keys as a list/CSV. - tests: Go asserts payment+stake adds a witness vs payment-only and that an unknown role errors; Python has the parity test. Rust/JS bindings verified to compile + load. - TODO.md marks the item done; docs/quicktx.md documents it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Converts "builds offline" into "a real node accepts it" for the straightforward intents: stake_registration, drep_registration, donation, governance info proposal, metadata, and Plutus mint. Each resets the devnet, funds the fixed test account, builds the intent's fixture with its real UTXOs, signs with the required key roles (e.g. payment+stake for staking, payment+drep for DRep, payment for the rest), submits, and asserts the tx is retrievable on-chain. Skips when DevKit is not running, so it runs only in the CI "Integration Tests (DevKit)" job. Native mint (no-key policy) and Plutus spend (lock-then-spend) follow once this batch confirms the pattern on CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First CI run surfaced real issues (all fixable): - stake_registration / metadata / Plutus mint already submitted fine; the on-chain check used a garbled hash (the devnet returns a chunked body, so devkitSubmitTx's "hash" was the chunk-size prefix). Verify via submit success (HTTP 200/202 = the node validated + accepted the tx) instead. - donation: the Conway cert asserts the stated treasury equals the chain's; regenerate the fixture with currentTreasuryValue=0 (fresh devnet). - drep_registration / governance_proposal: DevKit's /epochs/parameters omits drep_deposit / gov_action_deposit; inject them so the build can compute the certificate deposits (the node validates them on submit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
drep_registration / governance_proposal still failed: DevKit's /epochs/parameters returns drep_deposit and gov_action_deposit as null (present, not absent), so the if-absent guard skipped the injection. Set them unconditionally so the build can compute the certificate deposits; the node validates the values on submit. (stake_registration, metadata, Plutus mint, donation now pass on-chain.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5/6 now pass on-chain. The proposal failed with ProposalReturnAccountDoesNotExist: a Conway proposal's deposit-return account must be a registered stake address. Register it first, then submit the proposal in the next block. Refactor: extract devnetPP() (params + Conway deposits) and signSubmit() (build->sign->submit) so the proposal can run two sequential txs on one devnet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The minting fixture used a random policy key, so the account couldn't sign it for submission. Regenerate it under an empty ScriptAll policy (script_hex 820180) that requires no signature, and add TestIntegrationNativeMint which the fee payer alone submits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a plutus_lock fixture (payToContract: pays a UTXO to the always-succeeds script address carrying the datum hash) and TestIntegrationPlutusSpend, which: 1. locks 10 ADA at the script address, 2. finds the locked UTXO on-chain, 3. repoints the script_collect_from fixture's utxo_ref at it, and 4. spends it with the caller-supplied execution units. Completes the 8 straightforward submit-tests (the other 6 + native mint are green on-chain). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a submit-test per remaining concern, doing the on-chain setup each
certificate requires (sequenced txs on one devnet):
- voting_delegation : register stake -> delegate voting power (to abstain)
- drep_update : register DRep -> update DRep
- drep_deregistration : register DRep -> deregister DRep
- stake_withdrawal : register stake -> withdraw (zero) rewards
- stake_delegation : repoint the fixture at a real devnet pool, register+delegate
- voting : register DRep + stake -> submit info proposal (its build
tx hash is the gov action id) -> vote on it
- pool_registration : key the pool to the account's stake key (operator/owner/
reward account) so the stake-key signature witnesses it;
register the reward stake address first
Also inject pool_deposit (DevKit returns it null). Blind, CI-verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes (from the last run's ledger errors): - stake_withdrawal: Conway requires the stake address to be vote-delegated before it can withdraw, so register stake -> delegate voting -> withdraw. - stake_delegation: DevKit exposes no pool-list endpoint (/pools 404), so register a pool keyed to the account and delegate to it (the fixture is now delegate-only; the pool id is captured from StakePoolId). Depth (turn "node accepted" into "verifiably did the thing"): - native mint / Plutus mint now assert the minted asset is present at the receiver (assertMintedAssetAt). - Plutus spend asserts the locked script UTXO was actually consumed. - TestIntegrationDRepKeyRequired: a DRep registration signed with the payment key only must be rejected by the node, proving the extra sign_tx_with_keys witness is genuinely required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the bridge's bespoke intermediate JSON transaction format with CCL's native TxPlan (YAML) format, builds transactions fully offline, and verifies — both offline and against a real Cardano node — that the bridge produces node-acceptable transactions across the full intent surface (payments, staking, DRep, voting, governance, pools, metadata, and Plutus).
Net: ≈−13,200 lines (the bespoke spec, its mappers, the provider path, and the four per-language fluent builders are gone).
What changed
Core — TxPlan in, YAML out
QuickTxServicenow:TxPlan.from(yaml)→QuickTxBuilder(staticUtxoSupplier, () -> protocolParams, null)→compose(plan)→build()→ CBOR. Fully offline; never submits.ccl_quicktx_build(thread, yaml, utxos_json, protocol_params_json, exec_units_json). YAML in, YAML out; chain data is caller-supplied (no provider).Plutus script transactions (offline)
StaticTransactionEvaluatorto stamp them on — it never runs the script. Callers compute them with any evaluator (Ogmios/Blockfrost/Aiken/Scalus); seeTODO.md §2bfor the planned helper layer.Stake / DRep-key signing
ccl_account_sign_tx_multi(…, keys)signs with any subset ofpayment/stake/drep/committee_cold/committee_hot(CCLAccount.signWith*Key). Fixes theMissingVKeyWitnessesUTXOWrejection for stake/vote/DRep certs; the original payment-onlyccl_account_sign_txis unchanged. Wired through all wrappers (sign_tx_with_keys/SignTxWithKeys/signTxWithKeys).Wrappers — thin YAML pass-through
build(yaml, utxos, protocolParams, execUnits?)and parses the YAML result.Intent coverage — every intent, every wrapper, end-to-end
test-fixtures/quicktx-intents/: a shared, version-controlled fixture set (generated by the JVMQuickTxIntentsTestviaTxPlan.from(tx).toYaml()— CCL's exact intent shapes, single source of truth), incl. compose (multi-sender) and the Plutus mint/spend.On-chain validation (Yaci DevKit)
Decisions (agreed during development)
Verification
:core:test+:core:nativeCompilegreen; all four wrapper suites green (per-intent E2E + Plutus).Known limitations / follow-ups (not blockers)
TODO.md §2b): client-side Blockfrost/Ogmios/Aiken/Scalus helpers + examples for obtaining the units to pass in. (Deferred — the one remaining real feature.)