Question about handling non-conforming model responses

[blak0p]

Hi Giona, I’m Alejandro — opening this as an issue since there’s no Discussions tab enabled, not because anything’s broken.

I’m building something with a similar philosophy in a different domain (a Git MCP server where an LLM never decides semantics — a deterministic layer classifies changes first, the model only writes prose on top of that). Your typed-contract approach to model calls in tramAI hits the same nerve: trusting structure over raw LLM output.

I’m curious how you handle it when the model’s response doesn’t match the contract — do you retry, fail explicitly, or fall back to something else? Genuinely trying to compare notes, not just curious in the abstract.

[GionaGranchelli]

Hey! thanks for opening this, and no worries about using an issue. I should probably enable Discussions soon.

The short answer is: for structured outputs, TramAI does not silently coerce, guess, or return a partial object. It tries to repair the model response through a bounded retry loop, and if that still fails, it fails explicitly.

The flow today is roughly:

1. Generate a contract from the Kotlin return type

   For a non-"String" return type, TramAI asks the structured-output handler to create a schema-like JSON contract from the Kotlin type. That contract is then injected into the prompt with wording like:

   Respond only with valid JSON matching this schema:
   ...

   Relevant files:

   • "tramai-core/src/main/kotlin/dev/tramai/core/structured/StructuredOutputHandler.kt"
   • "tramai-core/src/main/kotlin/dev/tramai/core/structured/StructuredOutputContract.kt"
   • "tramai-engine/src/main/kotlin/dev/tramai/engine/TramaiEngine.kt"

2. Analyze the raw model response

   The Jackson implementation currently does three things:

   • extracts a JSON candidate from the raw response, including fenced JSON blocks;
   • deserializes that JSON into the target Kotlin type;
   • validates supported annotations such as "@AIRange" and "@AiMinItems".

   Relevant file:

   • "tramai-structured/src/main/kotlin/dev/tramai/structured/JacksonStructuredOutputHandler.kt"

3. Return either success or structured failure

   The handler returns a sealed "StructuredOutputResult":

   • "Success(value, rawResponse)"
   • "Failure(rawResponse, errorSummary, feedbackMessage)"

   The important bit is the "feedbackMessage". It is deliberately written in a form that can be appended back into the conversation, for example:

   Your previous response failed validation: Property 'confidence' must be between 0.0 and 1.0. Return corrected JSON only.

4. Retry with feedback

   The engine handles the failure by appending:

   • the model’s broken response as an assistant message;
   • the structured feedback as a user message.

   Then it calls the model again. The number of structured-output retries is controlled by "@operation(maxRetries)". The default is currently "2", so that means up to 3 total attempts including the first call.

5. Fail explicitly when retries are exhausted

   If the model still does not satisfy the contract, TramAI throws "StructuredOutputException". That exception preserves useful debugging context such as:

   • the original prompt;
   • the last raw response;
   • the validation/parsing error;
   • the total attempt count.

So the design is: show the model the contract first, give it specific repair feedback if it violates the contract, and fail loudly if it still cannot comply.

One nuance: TramAI does have provider retry/fallback behavior for provider/transport failures, but that is separate. A contract mismatch is not silently routed into a default value or “best effort” object. The caller either receives the typed return value or an explicit structured-output failure.

The tests that show this most directly are in:

• "tramai-structured/src/test/kotlin/dev/tramai/structured/JacksonStructuredOutputHandlerTest.kt"
• "tramai-engine/src/test/kotlin/dev/tramai/engine/TramaiEngineTest.kt"

In particular, there are tests for fenced JSON parsing, validation failure, retry after malformed output, and throwing "StructuredOutputException" after retries are exhausted.

Does my reply, answers your question?

[blak0p]

This is really clear, thanks for walking through the whole pipeline — the assistant/user replay for repair feedback makes a lot of sense as the actual self-correction mechanism.

Two follow-up questions on the contract side:

1. Is the JSON schema-like contract generated once and cached per call site, or rebuilt every time from the Kotlin type? Asking because if the return type evolves (e.g. a new field added), I’m curious whether that’s picked up automatically or if there’s a step to regenerate/version the contract.

2. Are @AIRange and @AIMinItems the full set of validators today, or is there an extension point for custom domain validators? Seems like that could get relevant fast once contracts represent more business-specific invariants beyond ranges/counts.

[GionaGranchelli]

Great questions.

On the contract generation side: today TramAI generates the contract from the Kotlin return type at runtime, through the configured "StructuredOutputHandler".

The current path is roughly:

TramaiEngine.executeStructured(...)
-> operation.structuredContract(handler)
-> handler.createContract(returnType)

With the default Jackson implementation, "createContract(...)" walks the Kotlin "KType" and produces the schema-like JSON that gets injected into the prompt.

So there is currently no separate generated contract file, no manual regeneration step, and no explicit contract version stored alongside the schema. The contract object is very small at the moment:

data class StructuredOutputContract(
val targetType: KType,
val schemaJson: String,
)

That means if the Kotlin return type changes, the schema generated by the running code changes with it. For example, adding a non-null field to the return DTO should cause the generated schema to include that field as required.

One related implementation detail: for cacheable structured operations, the schema is injected into the prompt before the cache key is built. So a changed schema should naturally change the prompt digest, rather than accidentally reusing a response for the old shape.

That said, there is no explicit “contract versioning” concept today. I think that is a real gap if TramAI grows toward longer-lived production agents, persisted workflows, or compatibility guarantees. A future improvement could be to expose a stable contract fingerprint/version, so applications can reason about schema drift more deliberately instead of relying only on the generated prompt content.

On validators: today the built-in validation set is intentionally small.

The default structured handler currently understands:

• "@AiDescription" — used as schema guidance for the model;
• "@AIRange" — runtime validation for numeric bounds;
• "@AiMinItems" — runtime validation for collection size.

So "@AIRange" and "@AiMinItems" are the real built-in runtime validators today. "@AiDescription" helps describe the contract to the model, but it is not a validator.

There is an extension point at the handler level: "StructuredOutputHandler" is an interface, and the engine receives it as a dependency. So in principle someone can replace the default Jackson handler with a custom implementation that generates and validates contracts differently.

But there is not yet a first-class custom validator API in the default handler. There is no current equivalent of:

registerValidator { ... }

or a small SPI for custom validation annotations.

I agree with your point that this becomes relevant quickly once the contract starts representing business invariants, not just basic structure. My current thinking is that TramAI should probably grow this in a deliberate way rather than hide domain rules inside prompts.

A clean future direction could be something like:

1. keep the current built-in annotations for simple structural constraints;
2. add a custom validation SPI for domain invariants;
3. make validation failures return the same kind of repair feedback used today;
4. optionally add contract fingerprints/versioning so persisted workflows can detect schema drift.

So the honest answer is: TramAI already has the typed contract + repair loop, but contract versioning and domain-specific validator registration are not first-class features yet. They are good candidates for the next layer.

Thanks for asking this, by the way. It made me realize that there are a couple of design gaps that I should probably make more explicit: contract versioning and domain-specific validation.
The current flow works for the simple case, but these are exactly the kind of edges that become important once people start using typed contracts in real workflows.

Thanks a lot!

[blak0p]

Thanks for the detailed breakdown, this makes a lot of sense.

On the contract versioning gap you mentioned: what would you think of a lightweight "contract fingerprint" — a short hash derived from the generated schemaJson (not the Kotlin type itself, since reflection on KType is more fragile to hash directly), stored alongside StructuredOutputContract? Two immediate uses I see:

1. Drift detection for persisted workflows: if the fingerprint stored when a workflow was created no longer matches the one generated by the current code, that's a clear signal something changed in the return type/schema, without needing to diff the full JSON.
2. A cheap comparison key for logging/observability, so you don't have to serialize and compare the whole schema on every check.

Does that fit with how you're thinking about evolving the handler, or do you see versioning living somewhere else — e.g. as an explicit annotation on the DTO itself rather than something derived at runtime?

[GionaGranchelli]

Yes exactly, I was thinking about something like a contract fingerprint derived from the generated schema Json, not from the Kotlin KType itself.
Something like

data class StructuredOutputContract(
    val targetType: KType,
    val schemaJson: String,
    val fingerprint: StructuredOutputContractFingerprint =
        StructuredOutputContractFingerprint.fromSchemaJson(schemaJson),
)

One thing I would probably avoid (at least at first) is making this a manually maintained DTO annotation only. An explicit annotation like @AiContractVersion("v2") could be useful later, but it has probably a different meaning.

I rather trust a sha256 fingerprint then a manually set annotation on a DTO.

But probably in the future we will need both of them, and for sure I'll focus first on the Json fingerprint. (It's already in my roadmap and I will tackle that soon)