feat(runtime): storage-write-dispatch@2.0.0 -- WIT + host trampolines#2
Open
zacharywhitley wants to merge 11 commits into
Open
feat(runtime): storage-write-dispatch@2.0.0 -- WIT + host trampolines#2zacharywhitley wants to merge 11 commits into
zacharywhitley wants to merge 11 commits into
Conversation
added 11 commits
June 25, 2026 08:03
…ing tests The WIT type contract grew 14 variants (int8/16/32, uint8/16/32, float32, date, time, timestamp, timestamptz, decimal, interval, uuid); the native bridge's duckvalue/logicaltype matches in engine.rs + reg_duckdb.rs were non-exhaustive and would fail to compile against the new contract (latent -- this crate isn't in the wasm smoke path). Add complete 1:1 mappings + a type_code widening map. Surfaced by cargo-mutants on reg_duckdb.rs: add tests that kill the safety-relevant survivors -- type_code_maps_every_logical_type, logical_type_maps_every_code, row_valid_reads_validity_bitmask (the NULL validity-mask predicate). The param_to_neutral per-type arms stay documented un-killed (duckdb::vtab::Value is an opaque FFI wrapper exercisable only e2e).
Add complex() arms in engine.rs + reg_duckdb.rs and make type_code by-ref for the now-variant Logicaltype, so the native bridge compiles against the escape-hatch contract. (This crate isn't in the wasm smoke path, so it must be checked explicitly on a types bump.)
De-risk spike for the native passthrough in the multi-provider architecture.
Q1 (LOAD hook): native DuckDB has no registerable seam to intercept a
transparent `LOAD <other>` -- PhysicalLoad::GetDataInternal calls
ExtensionHelper::LoadExternalExtension unconditionally and OnBeginExtensionLoad
returns void. The pragmatic mechanism is an explicit `ducklink_load('aba')`
entry (a C++/C-API extension can call the public ExtensionHelper::LoadExternal-
Extension / TryAutoLoadExtension itself); custom_extension_repository can serve
resolved native artifacts via autoinstall.
Q2 (dual-load): add passthrough::ducklink_load -- resolve a provider via the
lifted resolver, then load via the matching arm:
- wasm -> register_components (the Route-A wasmtime->C-API bridge, reused)
- native -> hand off to DuckDB's own `LOAD '<artifact>'`
- resolver.rs lifted verbatim from crates/ducklink-host (native stubs for
available()/trusted() enabled behind allow_native for the spike).
- bundled tests prove the wasm arm end-to-end (aba_validate in wasm) and that
the native arm is selected + dispatches to DuckDB's loader; native off by
default falls back to wasm.
- bump wasmtime 1ec8660(=39) -> 46.0.1 to match the local ducklink-runtime
(committed git pin had drifted); engine.rs error-context fixes for wasmtime 46.
…sign D)
Add aba_init_c_api: a DuckLink "shim" entrypoint named after a logical
extension. Stock DuckDB derives the init symbol from the filename
(<filebase>_init_c_api), so aba.duckdb_extension exports aba_init_c_api; on
LOAD aba it runs the multi-provider resolver in-process and dual-loads the
chosen provider (here the wasm aba via the proven bridge). Reuses passthrough.rs
(resolver + ducklink_load) verbatim; wasm path from DUCKLINK_ABA_WASM.
Proven end-to-end on the STOCK official v1.5.4 duckdb CLI (no core fork):
SET extension_directory=...; SET custom_extension_repository='<repo>';
INSTALL aba FROM '<repo>'; -- fetches the unsigned shim from a local repo
LOAD aba; -- shim resolves -> registers aba_validate (wasm)
SELECT aba_validate('021000021'); -- true, computed in the wasm component
And, after the one-time INSTALL, a fresh session's plain `LOAD aba` is fully
transparent (loads the persisted shim from the local extension dir).
Findings (stock duckdb, cited):
- custom_extension_repository does NOT make plain LOAD transparent for custom
names: AllowAutoInstall is a hardcoded 10-name list and even those use the
DEFAULT repo (extension_helper.cpp:144-158, extension_load.cpp:572-588). The
custom repo is consulted only by explicit INSTALL ... FROM.
- Transparency gradient: long-tail name = one-time INSTALL FROM, then plain LOAD
is transparent thereafter (persists to the local extension dir).
- Footer: C_STRUCT_UNSTABLE pins the exact engine version (v1.5.4); stable
C_STRUCT would be version-portable but the Rust C-API path needs unstable.
- Signing: requires -unsigned (startup-only) or a trusted-key-signed artifact.
… (design D)
Full first-cut of the transparent-LOAD passthrough on stock DuckDB.
Shim generator: `ducklink_shim!("<name>", <name>_init_c_api)` emits one shim
entrypoint per managed name (NAME is the only variable; body = shared
`passthrough::shim_load`). Provider selection is MANIFEST-DRIVEN: the shim reads
`DUCKLINK_HOME/index.json` (the generalized providers[]), resolves the best
certified provider, and dual-loads it -- NOT an env-forced provider. The
conformance HARD GATE is wired: a non-reference provider is admitted only if its
conformance.suite_digest == sha256(suite) (canonical) and at == wit_contract.
Native provider: `aba_native_init_c_api` registers a native-Rust `aba_validate`
(a real native .duckdb_extension served as aba_native.duckdb_extension). The
shim's native arm `LOAD`s it (re-entrant LOAD works on stock duckdb) -- the
native-speed passthrough.
Policy overrides are env-driven on stock duckdb (the stable C extension API has
no runtime-setting registration, so SET extension_provider can't be honored by a
C-API shim -- see docs/roadmap-to-signing.md): DUCKLINK_ALLOW_NATIVE,
DUCKLINK_PROVIDER, DUCKLINK_DENY.
tooling/: gen-shim.sh (footer-stamp a shim/native into the repo layout
<repo>/<ver>/<platform>/), ducklink-install.sh (INSTALL FROM -> extdir so plain
LOAD works), native-conformance.sh (run the suite vs the native provider, emit
the conformance record), verify-d.sh (the 4-scenario end-to-end proof).
Build: re-pin ducklink-runtime git rev fb4e228 -> 6f6b200 (origin/main, the
wasmtime-46 runtime) so the standalone repo / CI builds without the monorepo
path override; add sha2 for the suite content-digest.
Proven on the stock official v1.5.4 CLI (-unsigned): plain `LOAD aba` -> native
passthrough; force wasm; deny native -> wasm fallback; tampered suite_digest ->
native rejected by the gate -> wasm fallback; native conformance passes.
# Conflicts: # Cargo.lock # Cargo.toml # src/engine.rs # src/reg_duckdb.rs
…enario 4
native-conformance.sh now computes the canonical suite_digest with build C's
structured scheme over conformance.{sql,expected} (byte-identical to
resolver::compute_suite_digest), not a plain sha256, so the emitted record
matches the resolver's canonical. verify-d.sh adds scenario 4 (tampered
suite_digest -> gate rejects native -> wasm fallback) and runs over the unified
conformance suite.
…79 scaffold) Guest register_logical_type + register_cast calls succeed at load() and get captured into pending_logical_types / pending_casts, but engine.rs::load() dropped them on the floor when building LoadedComponent — only scalars/ tables/aggregates survived. This left GEOMETRY unregistered at DuckDB and every scalar taking GEOMETRY failed to bind: 'st_astext(BLOB) no match'. New LogicalTypeAlias + CastReg fields on LoadedComponent, populated from drain_pending output. reg_duckdb.rs gains register_logical_types_stub + register_casts_stub that log each intended registration. The stubs are no-ops today; the real DuckDB C-API wiring is the follow-up: 1. duckdb_create_logical_type(physical) -> handle 2. duckdb_logical_type_set_alias(handle, name) 3. duckdb_register_cast(source_handle, target_handle, callback) Once (1)-(3) land, ~213 currently-failing GEOMETRY-typed cases (#64/#67/#73 downstream of #79) should verify. This commit is the plumbing so the follow-up delta is minimal.
…gister_casts Replaces the stubs with real DuckDB C-API calls: register_logical_types: 1. duckdb_create_logical_type(physical) -> handle 2. duckdb_logical_type_set_alias(handle, name) 3. duckdb_register_logical_type(con, handle, null) 4. duckdb_destroy_logical_type(&mut handle) register_casts: 1. duckdb_create_cast_function() + set_source_type + set_target_type 2. duckdb_cast_function_set_implicit_cast_cost(cast, 0) # cheapest 3. duckdb_cast_function_set_function(cast, identity_cast_trampoline) 4. duckdb_register_cast_function(con, cast) 5. duckdb_destroy_cast_function + destroy_logical_type on locals Wired into direction-2 load pipeline BEFORE register_scalars so the binder sees GEOMETRY/GEOGRAPHY as valid types when scalars register their arg-type expectations. Casts follow with cost=0 for implicit resolution. identity_cast_trampoline: same-physical-type casts are byte-identical; the trampoline returns true without writing, and DuckDB's default row-copy propagates values verbatim. duckdb_type_of_physical: string→duckdb_type map covering BLOB/TEXT/INT64/ INT32/FLOAT64/FLOAT32/BOOLEAN with BLOB as safe fallback. Closes #79. Downstream #64 (GEOMETRY binder no-match) and #73 (st_makepoint demotion tracking) should resolve on the next batch: ~213 currently-failing GEOMETRY-typed corpus cases now have a bind path.
Previously the trampoline returned `true` without writing to the output vector, on the assumption that DuckDB would memcpy same-physical-type values by default. It does not — the callback must populate `output` itself. The empty output surfaced as a bare "Conversion Error" for BLOB->GEOMETRY (and the reverse) casts registered via #79. Iterate rows explicitly and copy validity + BLOB payloads via `duckdb_vector_assign_string_element_len`, mirroring the write path already used in write_ret_raw for BLOB returns.
Ports the writable-storage half from the ducklink-runtime workspace crate
(@4.0.0) into this SDK's runtime (@2.0.0). Additive within contract-major 2:
existing @2.0.0 read-only storage components keep loading unmodified; only
components that additionally export storage-write-dispatch instantiate the
new `duckdb-extension-storage-write` world.
WIT delta:
* new `storage-write-dispatch.wit` interface (7 methods: begin/commit/
rollback-transaction, create-table, insert-rows, delete-rows,
update-rows) mirroring the @4.0.0 shape verbatim, package header
rewritten to duckdb:extension@2.0.0.
* new `duckdb-extension-storage-write` world in the host WIT, exporting
storage-dispatch + storage-write-dispatch on top of the base imports.
* mirrored into wit-canonical/ so the witcanon digest reflects the
additive interface as part of the contract identity.
Runtime delta:
* new `duckdb_extension_storage_write_bindings` bindgen module; types
remapped via `with:` to the base bindings so the write trampolines
exchange `extension_types::*` directly (no per-world conversion).
* 7 new `storage_{begin,commit,rollback}_transaction` /
`storage_{create_table,insert,delete,update}_rows` methods on
`ExtensionInstance`, lazily building the writable-storage bindings
from the already-loaded raw instance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the writer half of the extension-runtime dispatch surface: the
storage-write-dispatch@2.0.0WIT interface plus the host-side trampolines innative-extension/ducklink/runtime/. Companion to the pre-existing read-side dispatch; symmetric shape with the sqlink write-dispatch work.Branch contents (stacked)
Eleven commits ahead of
main. They form a stacked sequence leading up to the write-dispatch trampolines and are cleanest to review together:f088df0feat(runtime): add storage-write-dispatch trampolines (contract 2.1.0) -- the headline change7dbf911fix(reg_duckdb): populate output vector in identity cast trampoline053cbe3feat(reg_duckdb): implement #79 C-API for register_logical_types + register_castsbad8ceafeat(engine): thread logical_types + casts through LoadedComponent (#79 scaffold)24c9268fix(tooling): conformance digest = C's structured scheme; verify-d scenario 4a04ac5dmerge: design-D shim + native arm onto re-vendored v0.2.0 runtime76ace70feat(shim): manifest-driven multi-provider shim + native aba provider (design D)7c816c2feat(shim): transparent LOAD via custom-repo shim on stock duckdb (design D)c9f5efbfeat(passthrough): native LOAD-hook + dual-load spike (design D)d4241fffeat(types): handle the complex (nested) duckvalue/logicaltype caseb776dbcfix(bridge): exhaustive duckvalue/logicaltype matches + mutation-killing testsIf reviewer prefers, the design-D shim commits and the #79 register-logical-types/casts commits could be split out into predecessor PRs -- happy to do that.
Related work
Motivating commits on the sqlink side (writer symmetry):
tegmentum/sqlink@5cfc37dc-- iVersion fixtegmentum/sqlink@6ec38652-- read/write unifyCompanion smoke-fixture repos:
tegmentum/synthetic-mutating-vtab-bridge(sqlink side)tegmentum/synthetic-mutating-vtab-ducklink-bridge(ducklink side)Companion PR:
tegmentum/ducklink#1-- adds the end-to-end write-boundary test in ducklink that exercises the trampolines added here.Test plan
cargo buildundernative-extension/ducklink/runtime/succeeds.tegmentum/ducklink#1write-boundary test passes when this branch is checked out intonative-extension/ducklink/.