Implementation Plan: Custom Codec Registry (Interface-Based Design)

[SeanTAllen]

This is the implementation plan for making the codec registry user-extensible. It builds on the design work from #147 and #146, and represents the output of an ensemble planning session with five different attention foci (API design, testing, correctness, security, completeness) that got synthesized and reviewed.

The big discovery during planning: the design in #147 assumed all 13 FieldDataTypes members would structurally conform to the new FieldData interface via Stringable. Turns out Array[U8] val doesn't have a string() method. It's not Stringable. So we need wrapper types, which makes this a bigger breaking change than the discussion anticipated.

The Core Idea

Separate result types from parameter types. The decode path gets an open interface (FieldData) that custom codecs can return their own types through. The encode path stays closed (FieldDataTypes union). This asymmetry is deliberate — parameters need to map to known PostgreSQL OIDs for the wire protocol, but results just need to be readable.

New Types

FieldData interface (postgres/field_data.pony)

interface val FieldData
  fun string(): String iso^

Open result type for decoded column values. 12 of the 13 built-in types already conform structurally via Stringable. The 13th — Array[U8] val — gets wrapped (see below). Custom codecs return their own types implementing this interface.

FieldDataEquatable interface (same file)

interface val FieldDataEquatable
  fun field_data_eq(that: FieldData box): Bool

Opt-in equality for custom FieldData types in Field.eq(). Built-in types use explicit match arms as they do today. Custom types implement this to participate in field equality comparisons.

Bytea (postgres/bytea.pony)

class val Bytea is (FieldData & Equatable[Bytea]) — wraps Array[U8] val for PostgreSQL bytea columns (OID 17). Public let data: Array[U8] val field. eq() does byte-by-byte comparison. string() returns hex representation.

RawBytes (postgres/raw_bytes.pony)

class val RawBytes is (FieldData & Equatable[RawBytes]) — wraps Array[U8] val for unknown binary-format OIDs where no codec is registered. Same API shape as Bytea but semantically distinct: Bytea means "this is known binary data from a bytea column," RawBytes means "we don't know what this OID is and here are the raw bytes."

Changes to Existing Types

Codec.decode() (postgres/codec.pony)

Return type changes from FieldDataTypes ? to FieldData ?. The encode method stays FieldDataTypes. Docstring explains the asymmetry.

Built-in codecs (_binary_codecs.pony, _text_codecs.pony, _text_passthrough_binary_codec.pony)

Mechanical: each decode return type becomes FieldData. Bodies unchanged except _ByteaBinaryCodec.decode() returns Bytea(data) instead of raw data, and _ByteaTextCodec.decode() returns Bytea(decoded_bytes) instead of the raw array. All encode signatures unchanged.

CodecRegistry (postgres/codec_registry.pony)

1. decode() return type: FieldDataTypes → FieldData
2. Binary fallback: raw data → RawBytes(data)
3. Text fallback unchanged (String.from_array(data) already conforms)
4. New public method for fluent custom codec registration:

fun val with_codec(oid: U32, codec: Codec): CodecRegistry =>
  CodecRegistry._with_codec(this, oid, codec)

Delegates to the existing _with_codec constructor. Supports chaining: CodecRegistry.with_codec(600, A).with_codec(790, B).

Field (postgres/field.pony)

let value: FieldData (was FieldDataTypes). The eq() match gets two new arms replacing Array[U8] val:

| (let a: Bytea, let b: Bytea) => a == b
| (let a: RawBytes, let b: RawBytes) => a == b

Plus a FieldDataEquatable arm before else for custom types:

| (let a: FieldDataEquatable, let b: FieldData) =>
  a.field_data_eq(b)

_RowsBuilder (postgres/rows.pony)

let field_value: FieldData (was FieldDataTypes).

Threading CodecRegistry Through the State Machine

The registry needs to flow from Session.create() down to _SessionLoggedIn where it's used. This follows the existing threading pattern for _notify and _database_connect_info.

Session.create() gets an optional parameter with CodecRegistry as default:

new create(
  server_connect_info': ServerConnectInfo,
  database_connect_info': DatabaseConnectInfo,
  notify': SessionStatusNotify,
  registry: CodecRegistry = CodecRegistry)

Call site: Session(server_info, db_info, notify where registry = my_registry)

The registry threads through the full chain: _SessionUnopened → _ConnectableState → _SessionSSLNegotiating / _SessionConnected → _AuthenticableState → _SessionSCRAMAuthenticating → _SessionLoggedIn. Each state class gets a let _codec_registry: CodecRegistry field and constructor param. The _ConnectableState and _AuthenticableState traits get an abstract fun codec_registry(): CodecRegistry method that concrete states implement.

_SessionLoggedIn changes from creating its own CodecRegistry internally to receiving one via constructor param.

Tests

Updated existing tests

_test_codecs.pony: assert_is[FieldDataTypes] → assert_is[FieldData] (~6 occurrences). Bytea codec tests match on Bytea and use .data for byte-level assertions. _TestCodecRegistryDecodeUnknownBinary matches RawBytes instead of Array[U8] val.

_test_equality.pony: Array[U8] val field values → Bytea in example-based tests. Generator renamed _FieldDataTypesGen → _FieldDataGen with output Generator[FieldData]. Property test type params updated throughout.

_test_query.pony: _ByteaQueryReceiver matches Bytea instead of Array[U8] val, uses .data for assertions.

_test_session.pony: _TestByteaResultDecoding receiver matches Bytea instead of Array[U8] val.

New tests

Wrapper type tests: _TestByteaString (hex output), _TestByteaEquality (equal, different lengths, empty), _TestRawBytesString, _TestRawBytesEquality.

Custom codec tests: _TestCodecRegistryWithCodecBinary, _TestCodecRegistryWithCodecText, _TestCodecRegistryWithCodecOverride, _TestCodecRegistryWithCodecChaining, _TestCodecRegistryWithCodecPreservesBuiltins, _TestRowsBuilderWithCustomCodec. Test helpers include a _TestPoint val class implementing FieldData + FieldDataEquatable and a _TestPointCodec primitive.

Custom equality tests: _TestFieldEqualityCustomType, _TestFieldInequalityCustomType, _TestFieldEqualityCustomVsBuiltin, _TestFieldEqualityCustomWithoutEquatable, _TestFieldCustomEqualityReflexiveProperty.

Examples

Seven existing examples updated: bytea, crud, named-prepared-query, prepared-query, query, ssl-query, ssl-preferred-query. Change | let v: Array[U8] val => to | let v: Bytea => and use v.data for byte access. Copy-in/copy-out examples unchanged (they use Array[U8] val for COPY wire data, not field values).

New custom-codec example: custom Point class implementing FieldData + FieldDataEquatable, binary codec for PostgreSQL point (OID 600), registry via CodecRegistry.with_codec(600, PointBinaryCodec), passed to Session with where registry = ..., matching on Point in results.

Documentation and Release Notes

• Package docstring (postgres/postgres.pony): new "Custom Codecs" section, updated "Working with Results" with Bytea instead of Array[U8] val
• Stale references in .release-notes/next-release.md updated to Bytea
• New release notes file documenting the two breaking paths: (1) bytea results change from Array[U8] val to Bytea, (2) unknown binary OIDs change from Array[U8] val to RawBytes
• CLAUDE.md updated with new types and changed APIs
• examples/README.md updated with custom-codec entry

Classification

This is a changed (breaking) change. The PR gets the changelog - changed label. Two breaking paths for users: matching on Array[U8] val in result fields needs to become Bytea (or RawBytes for unknown binary OIDs).

[SeanTAllen]

Implemented in #150