@clarit.ai/vercel-ai-provider

Vercel AI SDK models that remember.

Standard LLM providers treat model state as disposable. Every new message re-reads the entire conversation history from scratch, costing you time and tokens.

Clarit (Engram) changes the rules.

This provider for the Vercel AI SDK turns model state into a durable asset. Save conversation state to disk and restore it in ~2ms. Skip the prefill, slash token costs by 94%, and pick up exactly where you left off.

---

Installation

npm install @clarit.ai/vercel-ai-provider ai

Features

• Drop-in Persistence: Wrap any OpenAI-compatible Mamba/hybrid model.
• Auto-Save Snapshots: Automatically persist conversation state to the Engram 3-tier hierarchy (VRAM/RAM/Disk).
• Restore-and-Generate: Perform inference by restoring state directly, bypassing standard /v1/chat/completions prefill overhead.
• Snapshot Management: Direct TypeScript client for listing, querying, and deleting conversation checkpoints.

Quick Start

import { createClarit } from '@clarit.ai/vercel-ai-provider';
import { generateText } from 'ai';

const clarit = createClarit({
  baseURL: 'http://your-engram-server:30000',
});

// 1. Initial Turn: Save the context
const { text } = await generateText({
  model: clarit('granite-4.0-h-small'),
  messages: [{ role: 'user', content: 'My name is Alice.' }],
  providerOptions: {
    clarit: {
      conversationId: 'session-123',
      turnNumber: 1,
      autoSaveSnapshot: true,
    },
  },
});

// 2. Future Session: Instant Recall
// Restores Alice's state in ~2ms, skipping re-reading Turn 1.
const { text: recalled } = await generateText({
  model: clarit('granite-4.0-h-small'),
  messages: [], // History is restored from state, not re-read from text
  providerOptions: {
    clarit: {
      conversationId: 'session-123',
      turnNumber: 1,
      restoreBeforeGenerate: true,
      continuationIds: [5812, 1072, 30], // Tokenized "What is my name?"
      maxNewTokens: 64,
    },
  },
});

Provider Options

Option	Type	Description
conversationId	string	Stable ID for grouping conversation snapshots.
turnNumber	number	The turn index to save to or restore from.
branchName	string	Alternate conversation branch name.
autoSaveSnapshot	boolean	Save state after generation completes (best-effort).
restoreBeforeGenerate	boolean	Mode: Restore state -> Append tokens -> Generate.
continuationIds	number[]	Token IDs for the new turn (required for restore mode).
maxNewTokens	number	Max tokens to generate in restore mode.
compatibilityMode	'append-only' | false	Enable stateless compatibility bridge (@experimental).

Compatibility Mode (@experimental)

For chat harnesses that don't natively manage Engram state, the append-only compatibility bridge automatically detects when new messages are an extension of a previously snapshotted conversation and takes the fast-path (restore-and-generate), falling back to standard /v1/chat/completions otherwise.

import { type ClaritResponseMetadata } from '@clarit.ai/vercel-ai-provider';

const { text, providerMetadata } = await generateText({
  model: clarit('granite-4.0-h-small'),
  messages: [
    { role: 'system', content: 'You are helpful.' },
    { role: 'user', content: 'Hello!' },
    { role: 'assistant', content: 'Hi there!' },
    { role: 'user', content: 'What is 2+2?' },  // ← new message appended
  ],
  providerOptions: {
    clarit: {
      conversationId: 'session-123',
      autoSaveSnapshot: true,
      compatibilityMode: 'append-only',
      maxNewTokens: 128,  // required for fast-path
    },
  },
});

// Check what happened
const meta = providerMetadata?.clarit as ClaritResponseMetadata;
console.log(meta.compatibilityResult); // 'fast-path' | 'fallback'

How it works

1. Tokenizes the full message list via the server's /tokenize_chat endpoint.
2. Fetches the latest snapshot's fill_ids (cached tokens) via /get_snapshot_info.
3. Checks whether the new messages are a pure append on top of the snapshot using isAppendOnly().
4. Fast-path: If append-only, restores the snapshot and sends only the delta tokens — skips re-prefilling the entire conversation.
5. Fallback: If the conversation was edited, contains unsupported content, or no snapshot exists, falls back to standard generation.

Fallback reasons

Reason	Meaning
disabled	maxNewTokens not set or zero.
no-snapshot	No snapshot found for the conversation.
fill-ids-missing	Snapshot exists but has no fill_ids.
prefix-mismatch	Messages were edited (not an append-only extension).
unsupported-content	Prompt contains tool calls, files, or other non-text content.
tokenization-error	Server-side tokenization failed.
missing-conversation-id	No conversationId provided.

Response metadata

When compatibility mode is active, providerMetadata.clarit includes diagnostics:

Field	Description
compatibilityResult	'fast-path' or 'fallback'
fallbackReason	Why fallback was taken (see table above)
reusedTokenCount	Tokens reused from snapshot (fast-path only)
continuationTokenCount	New delta tokens sent (fast-path only)

Snapshot Management

Access the Engram management API directly via clarit.snapshots:

const snapshots = await clarit.snapshots.list('conversation-123');
console.log(`Found ${snapshots.length} checkpoints.`);

await clarit.snapshots.delete({
  conversation_id: 'conversation-123',
  turn_number: 1
});

Built-in Agent Skills

This repo ships Claude Code / agent skills in .claude/skills/ that give AI assistants working knowledge of the systems this provider touches — no extra setup required:

Skill	Purpose
vercel-ai-provider-spec	Vercel AI SDK Language Model spec (V3/V4), type reference, and upgrade patterns
engram-sglang	Engram server architecture, snapshot API contract, server flags, model compatibility matrix

If you're using Claude Code, these are loaded automatically. For other agents, the Markdown content in each SKILL.md and its references/ directory is designed to be consumable as-is.

License

Apache-2.0