RFC 0001 — Typed, discoverable resources (+ code-first schema spikes)

[dawsontoth]

Status: Draft — RFC for discussion; the spikes have now been promoted into a working implementation of the canonical code-first model (phase 2b), landing on this branch.

What this is

RFC + implementation for strengthening type discoverability and strength across Harper resources. The full RFC is in docs/rfcs/0001-typed-discoverable-resources.md.

The anchoring principle (forced by --conditions=typestrip): runtime metadata must be values; TypeScript types are derived from them — never the reverse. Taken to its conclusion, the schema itself can be authored as a TS value with the record type inferred.

Three pillars

1. One canonical schema model — define tables in GraphQL or TypeScript; both compile to one internal model, projected to every surface from a single core (attributeToFragment).
2. A per-method request contract — withSchema/defineResource carrying params/query/body/response, with handler types derived and the same declaration feeding validation + OpenAPI + MCP.
3. Typed base-request behavior — base types for Harper's built-in filter/sort/select/relationship-include grammar, keyed to the record and its relationship graph.

Implementation: canonical defineTable + types (NEW — resources/defineTable.ts)

The canonical-track model, for real:

import { defineTable, types } from 'harperdb';
const { id, int, string, date } = types;

export const Track = defineTable('Track', {
	id: id.primaryKey,                                   // getter-style flags
	name: string.indexed,
	duration: int.nullable,
	status: types.enum(['draft', 'published']).indexed,
	createdAt: date.createdTime,
});
// Track IS the live registered class — the import is the typed handle
type Track = InstanceType<typeof Track>;
type InsertTrack = (typeof Track)['$insert'];           // $record/$insert/$upsert/$patch/$query

• Eager registration through the canonical path: defineTable compiles to the same typeDef GraphQL builds and registers via the existing table() factory — shared DDL/migration semantics (re-definition re-asserts/evolves the schema), not a parallel path.
• One name, projected per verb: singular Track is the table you query/write through and the instance type; $record (readonly reads), $insert (PK optional), $upsert (PK required), $patch, $query (indexed fields only) are inferred members, never parallel names.
• Relations via lazy thunks: types.relation(() => Album, { from: 'albumId' }) / types.hasMany(...) — forward references and cycles are safe (the definition late-binding contract the query-time resolvers already rely on). Like GraphQL, the to-one FK must be declared in the shape, which makes it typed, projected, and writable.
• GraphQL parity locked by test: both front-ends project identical canonical properties per field (including the nullability mapping: code-first plain field ≡ !, .nullable ≡ GraphQL plain; PK/server-managed unmarked).
• Loader untouched: exporting a handle from resources.js auto-exposes it through the existing resource walk — same semantics as export class X extends tables.X {}.
• Type contract enforced on the real module: docs/rfcs/spikes/0001/defineTable-real.check.ts re-runs the canonical-track assertions (incl. @ts-expect-error negatives) against the built package types.
• relationOf/hasManyOf — mutual-pair escape hatch: a truly mutual pair (Book.author <-> Author.books, each referencing the other) breaks TS's eager const-initializer inference — typeof Book/typeof Author each depend on the other and collapse to any (TS7022), verified empirically both ways (fails with plain relation/hasMany on both sides, passes with the escape hatch). relationOf<R>/hasManyOf<R> type the thunk target () => any and take the related record shape explicitly via <R> instead of inferring it, breaking the cycle; runtime is identical to relation/hasMany (only the type path differs — proven by a dedicated registration test).

Verify: npm run build && npx mocha unitTests/resources/defineTable-registration.test.js unitTests/resources/jsResource.test.js → 16 passing; npx tsc --noEmit --project docs/rfcs/spikes/0001/tsconfig.json → green.

Cross-model review (Codex + Gemini + Harper-domain, adjudicated)

No blockers. Addressed in-branch: OpenAPI $ref components for code-first relations now always resolvable (lazy type/attributes on the relation definition); to-one FK must be declared (was an untyped auto-added column). Surfaced for discussion:

• Open design question — explicit null: nullable fields type as ?: T (absence), but the runtime accepts/returns explicit null for unmarked-nullable fields, so patch(id, { duration: null }) is runtime-valid yet type-rejected. Drizzle-style would be T | null. The canonical spike pins optional-only; keeping it that way unless we decide otherwise.
• Follow-up — enum enforcement: types.enum() narrows the TS type, but the shared Table.validate() has no enum case (true for GraphQL today too) and attributeToFragment drops the constraint. A runtime enum check would benefit both front-ends; deferred as its own issue.

Spikes (proof-of-concept, superseded by the implementation above)

• (a) RFC document — docs/rfcs/0001-typed-discoverable-resources.md
• (b) t builder + defineTable inference — docs/rfcs/spikes/0001/t-builder.spike.ts (superseded by canonical-track)
• (c) defineTable value → table() registration — promoted to unitTests/resources/defineTable-registration.test.js, now exercising the real public API
• canonical-track — one Track, every verb shape inferred — docs/rfcs/spikes/0001/canonical-track.spike.ts (the model the implementation follows)
• enforceSchema — subset-typed request contract — docs/rfcs/spikes/0001/enforce-schema.spike.ts (next up: its record/body slots take the $ projections directly)

This PR is kept up to date as work proceeds. Not for merge as-is — it is a living design surface.

Context

Grounded in a review of harper core (resources/, components/mcp/, server/) and the HarperFast/schema-codegen plugin. Verified findings (e.g. parameterized custom resources produce zero MCP tools; OpenAPI emits none of the query grammar; four divergent type translators over one Attribute) are tabulated in §11 of the RFC.

🤖 Generated with Claude Code

[claude]

Reviewed; no blockers found.

[gemini-code-assist]

Code Review

This pull request introduces RFC 0001, which proposes a unified approach for typed, discoverable resources in HarperDB through a single canonical schema model, per-method request contracts, and typed query behaviors. The feedback suggests improving the proposed PathParams utility type to support wildcard path segments (e.g., *filepath) to align with the existing runtime capabilities.

[kriszyp]

This is excellent, lots of great ideas in here. Let me comment on some of them:

One source of truth ... Today GraphQL SDL is the only way to define a table, and the canonical Attribute[]

Yes, that was previously true, and we did have a lot of muddiness with this in the past, but I believe #1167 has really solidified a single source of truth around canonical JSON-schema aligned data structures on classes. This is fed from graphql and is compatible/consistent across tables and custom resources, I believe, and largely addressed pillar 1.

defineTable

I think defineTable would be a great addition. I think there are some things to consider here:

1. I would like to consider a little more literate version of this, is it feasible to write:

import { defineTable, types } from 'harper';
const { id, int, string } = types;
export const Tracks = defineTable('Tracks', {
	id: id.primaryKey,
	name: string.indexed,
	duration: int.nullable,

2. There are numerous options for tables beyond their fields, including expiration, sealed, replicated, etc. I guess that all goes in a third arg?
3. I still believe that trying to teach people how to simultaneously deal with a singular Track and plural Tracks is going to be confusing. I have very intentionally sought a singular table name and type to give devs a single concept work with. I think when/why to plurarize is not a complexity that we want to burden devs with.

4.2 Serve every surface from one projector

Is one projector different than a single source of truth?

Pillar 2 — A per-method request contract

Augmenting a resource with schema-enforcement (and trustable types built on that), sounds good. But the current withSchema seems rather hostile to our best-practice of static methods and seems to introduce a third request/target type, RequestOf. I don't think we want the complexity of third request type, I think polymorphic/type-subset compatibility should be a explicit goal of schema-ification. And it should be compatible with top-level (non-instance functions):

export const Widget = {
	path: '/widget/:id',
	async get(req) {
		return loadWidget(req.id, req.get('expand'));
	}
	async post(req) {
		return createWidget(req.body);
	}
}
// and StricterWidget should be type compatible stricter subset, directly drop-in replacement, no new types
export const StricterWidget = enforceSchema<RecordType>({ // RecordType is default return type of `get` and input type of `put`
	get: ... extra type definitions
	post: ...
// req.id: string · req.get('expand')?: ('parts'|'owner')[] * req.body all are now inferred in a type subsetting fashion
}, Widget);

I'm hesitant to get into too much discussion of later pillars since they appear to somewhat build on top of each other.

Also, I'm curious about the intended future of the PR. Is 0001-typed-discoverable-resources.md a transient component of the PR? We have intentionally been putting proposals in GH issues (and several much larger than this) rather than PRs to avoid transient information in the repo. However, implementation plans as a transient artifacts of a PR in progress is something we have occasionally done.

[dawsontoth]

@kriszyp breaking the various threads out for easier discussion:

One source of truth ... Today GraphQL SDL is the only way to define a table, and the canonical Attribute[]

Yes, that was previously true, and we did have a lot of muddiness with this in the past, but I believe #1167 has really solidified a single source of truth around canonical JSON-schema aligned data structures on classes. This is fed from graphql and is compatible/consistent across tables and custom resources, I believe, and largely addressed pillar 1.

I think 1167 helps to fill out the information we can feed out through MCP and OpenAPI, but it doesn't tackle the harder problem I see pillar 1 addressing: a single, discoverable, cohesive surface defining the schemas that infers types with full language support without any generation.

[kriszyp]

So a canonical schema encompasses the scope of types? That seems to contradict pillar's assertion that it "also be authored in TypeScript", and "no generation"; i.e. we aren't attempting to create (canonical)types from graphql in the scope of this PR, are we? However, I definitely support the ability to define a single canonical source of type definitions and runtime metadata for a specific table from a defineTable call though (that's just don't make everything have a universal single source of truth across types and runtime).

[dawsontoth]

I would like to consider a little more literate version of this, is it feasible to write:

class Types {
	private metadata: Partial<Record<keyof Types, any>> = {};

	get id(): Types {
		this.metadata.id = true;
		return this;
	}
	get string(): Types {
		this.metadata.string = true;
		return this;
	}
	enum(values: string[]): Types {
		this.metadata.enum = values;
		return this;
	}
	get indexed(): Types {
		this.metadata.indexed = true;
		return this;
	}
	get nullable(): Types {
		this.metadata.nullable = true;
		return this;
	}

	build(): Partial<Record<keyof Types, boolean>> {
		return this.metadata;
	}
}


const types = new Types();

console.log('empty build:');
console.log(types.build());

console.log('example string');
console.log(types.string.indexed.build());

console.log('example enum');
console.log(types.indexed.enum(['a', 'b']).build());

output:

empty build:
{}
example string
{ string: true, indexed: true }
example enum
{ string: true, indexed: true, enum: [ 'a', 'b' ] }

[kriszyp]

Seems like you could do this without the build() call (by skipping metadata layer), right?

[dawsontoth]

I still believe that trying to teach people how to simultaneously deal with a singular Track and plural Tracks is going to be confusing. I have very intentionally sought a singular table name and type to give devs a single concept work with. I think when/why to plurarize is not a complexity that we want to burden devs with.

@kriszyp I think this shows where the singular vs plural discussion comes to something fruitful. A table doesn't give or receive a single interface, it needs different things when you're querying, responding, inserting, upserting, patching, etc. By having the table class in TypeScript, we can follow the type inference precedents set out by other libraries to make it really easy to maintain this sprawl of types. Because the fields are defined in TypeScript, we can include or omit fields automatically.

[kriszyp]

We can't genericize t.Select<typeof Track> and t.Insert<typeof Track?

[dawsontoth]

Can you show me what you mean with example code? I'm a bit confused.

[kriszyp]

here is what I generated for a spike/example: https://github.com/HarperFast/harper/blob/claude/sad-lehmann-1b3b70/docs/rfcs/spikes/0001/canonical-track.spike.ts

[dawsontoth]

sure, looks good to me

[dawsontoth]

There are numerous options for tables beyond their fields, including expiration, sealed, replicated, etc. I guess that all goes in a third arg?

I think so. Another approach would be to shift to accept a single dictionary and do it based off of names, nest the fields in one level. That would be more discoverable, and I like the visual nesting you get, too.

[dawsontoth]

Also, I'm curious about the intended future of the PR. Is 0001-typed-discoverable-resources.md a transient component of the PR? We have intentionally been putting proposals in GH issues (and several much larger than this) rather than PRs to avoid transient information in the repo. However, implementation plans as a transient artifacts of a PR in progress is something we have occasionally done.

It was purely because PRs have threads, separate files, and syntax highlighting. All of which feel very helpful for the conversation. I don't think the RFC would actually land in the repo, since over time it'll become inaccurate. The 0001 was Claude being optimistic.

[dawsontoth]

Is one projector different than a single source of truth?

I don't know that projector was the best choice of words, but the way I was visualizing it is... GraphQL can be a single source of truth, yes. But there isn't language-service-level-support for making it project outward into JavaScript and TypeScript in all the ways we need for a great developer experience. That's where the codegen plugin I wrote comes in, but it has compromises (like causing a cascade of Harper restarts, or only doing stuff when Harper is running it). OpenAPI and MCP do similar work for us because GraphQL doesn't natively have a bridge to TS/JS.

By instead shifting the source of truth to be in that TS/JS world, as soon as the characters are typed, the interfaces can be inferred without any codegen (thus eliminating that plugin). The MCP and OpenAPI work becomes more succinct as well.

[dawsontoth]

Augmenting a resource with schema-enforcement (and trustable types built on that), sounds good. But the current withSchema seems rather hostile to our best-practice of static methods and seems to introduce a third request/target type, RequestOf. I don't think we want the complexity of third request type, I think polymorphic/type-subset compatibility should be a explicit goal of schema-ification. And it should be compatible with top-level (non-instance functions):

export const Widget = {

Couldn't we use the provided language concepts for polymorphism? The struggle with statics is discoverability and adherence to an interface. You can put anything you want on export const Widget, and if you typo gwt instead of get, we have no mechanism to detect if this is a messed up resource or table or some utility class that we should ignore. But yes, this does lead to a very different interface across the stack, and hopefully to the removal of the old interfaces in a large breaking change, v6, v7, whenever.

[kriszyp]

@dawsontoth I put together spikes at docs/rfcs/spikes/0001/t-builder.spike.ts and an enforce-schema spike to plan out these paths. Let me know what you think (and I could proceed with implementation).

[dawsontoth]

Could these collapse together into a single get and post, instead of having two positional arguments? So...

get: {
	query: { expand: s.arrayOf(s.enumOf(['parts', 'owner'])).optional },
	execute: async get(target) {
		// target.id: string (from ':id') · target.get('expand'): ('parts'|'owner')[] | undefined
		return loadWidget(target.id, target.get('expand'));
	},
}

[dawsontoth]

Nice