Custom Codec Registry: Phase 3 Design

[SeanTAllen]

Custom Codec Registry: Phase 3 Design

Design for Phase 3 of the codec work (discussion #139). Phases 1 (binary parameters, PR #141) and 2 (binary results, PR #144) are complete. Phase 3 makes the codec registry user-extensible so users can register custom codecs for PostgreSQL types not covered by built-ins.

The FieldDataTypes Extensibility Problem

The original Phase 3 plan (in #139) assumed custom codecs would "just work" by making _with_codec public and adding a CodecRegistry parameter to Session. But there's a fundamental tension: FieldDataTypes is a closed union type.

type FieldDataTypes is
  ( Array[U8] val | Bool | F32 | F64 | I16 | I32 | I64
  | None | PgDate | PgInterval | PgTime | PgTimestamp | String )

Codec.decode() returns FieldDataTypes. A custom codec for PostgreSQL point (OID 600) can only return one of these existing variants — it can't return a custom Point class. The best it can do is decode to String (e.g., "(1.5,2.3)") or Array[U8] val. Same limitation on the encode side: Codec.encode() takes FieldDataTypes, so users can't send custom Pony types as parameters.

This means custom codecs without FieldDataTypes extensibility have narrow value:

1. Decode unknown OIDs to String instead of opaque Array[U8] val
2. Override built-in codec behavior for existing types
3. Foundation for future expansion

For comparison, other drivers solve this with open-world type systems:

• Go's pgx: any (interface{})
• Rust's tokio-postgres: FromSql/ToSql trait objects
• Python's psycopg3: dynamic typing
• Elixir's Postgrex: protocols

Pony's type system makes this harder. The options:

Option A: Add a wrapper variant to the union

class val CustomFieldData
  let inner: Any val

type FieldDataTypes is
  ( Array[U8] val | Bool | F32 | F64 | I16 | I32 | I64
  | None | PgDate | PgInterval | PgTime | PgTimestamp | String
  | CustomFieldData )

Users match on CustomFieldData and cast the inner value. The union stays closed (one more variant), exhaustive matching still works. But the inner value requires runtime type knowledge — users need to know what their custom codec put in there. Breaking change: all existing exhaustive match expressions on FieldDataTypes need a new arm.

Option B: Keep FieldDataTypes closed, accept the limitation

Custom codecs can only decode to existing variants. Phase 3's value is "decode unknown OIDs to readable strings instead of opaque bytes" plus overriding built-in codecs. Simpler, no breaking change beyond making the API public.

Option C: Defer Phase 3 entirely

If the narrow scope isn't worth shipping and the open-world design needs more thought, park it. The internal _with_codec infrastructure is already there for when the design is ready. Ship Phase 3 when there's a concrete use case driving the requirements.

The encode Method Question

The original plan adds fun encode(oid: U32, value: FieldDataTypes): Array[U8] val ? to CodecRegistry. But the current encode path in _FrontendMessage.bind() matches on FieldDataTypes variants (Pony types) to select encoding, not on OIDs. A registry-based encode lookup by OID doesn't fit the architecture without refactoring.

Questions:

• Is encode on the registry needed for Phase 3, or can it be deferred?
• If custom types can appear as parameters (Option A with CustomFieldData), how does the bind path know which codec to use? The OID isn't derivable from the Pony type — it would need to be specified somehow.
• Should parameter encoding for custom types be a separate concern from result decoding?

Phase 3 Scope (from #139 v3 plan)

For reference, the original Phase 3 steps:

1. Make CodecRegistry user-configurable — rename _with_codec to with_codec (public), add encode method
2. Add optional registry parameter to Session constructor
3. Custom codec documentation in package docstring
4. New examples/custom-codec/ example + update examples/README.md
5. Document FieldDataTypes expansion cost
6. Tests, CLAUDE.md update, release notes (classification: added)

Design Questions

1. Should FieldDataTypes be extended with a CustomFieldData wrapper (Option A), kept closed (Option B), or should Phase 3 be deferred (Option C)?
2. If Option A: what should CustomFieldData wrap? Any val is the most flexible but requires runtime casting. A trait with string(): String would give basic formatting.
3. Should encode on CodecRegistry be part of Phase 3 or deferred? The decode side (results) is straightforward; the encode side (parameters) has architectural friction.
4. If custom parameter encoding is supported, how does the user specify the OID? The current architecture derives OID from the Pony type, which doesn't work for custom types.

[SeanTAllen]

replaced with #147