Skip to content

chore: release 0.8.2 #180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rlippmann merged 5 commits into from
Jul 10, 2026
Merged

chore: release 0.8.2 #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
250ab1b
docs: improve companion repository discoverability
rlippmann Jul 10, 2026
26da8a0
docs: extract public API reference
rlippmann Jul 10, 2026
a73e2f3
docs: streamline top-level README
rlippmann Jul 10, 2026
0fd4717
docs: clarify examples and demos
rlippmann Jul 10, 2026
a4de123
chore: release 0.8.2
rlippmann Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 55 additions & 146 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,15 @@ Context Compiler gives hosts fixed state rules:
- clarification instead of silent overwrite for blocked/ambiguous changes
- pending confirmation flows that must resolve before anything else changes
- export and import checkpoints to restore saved state and pending confirmation flow
- produce structured saved state that the host can pass to the model
- produce structured authoritative state for downstream host decisions

The model generates responses. The compiler owns state transitions.
The model generates responses. The compiler owns state.

## How the compiler metaphor works

Context Compiler treats important instructions as structured state instead of
temporary prompt text.

Like a compiler, it parses input, validates it, applies fixed rules, and
produces a stable result the host can use. It is not source-code
produces a stable result the host can use. It treats important instructions as
structured state instead of temporary prompt text. It is not source-code
compilation and not a reasoning model.

## 10-Second Example
Expand Down Expand Up @@ -98,8 +96,8 @@ Host Application
└─ update → authoritative state mutated; host may call LLM with compiled state
```

The compiler updates state and never calls the LLM.
Your app decides whether to call the model based on the returned `Decision`.
The compiler never calls the LLM. Your app decides what to do with the returned
`Decision`.

---

Expand Down Expand Up @@ -129,8 +127,15 @@ else:
render(call_llm(user_input))
```

This is the main integration path: your app owns the model call, and the
compiler owns deterministic state transitions.
This is the main integration path: your app owns the model call and uses the
compiler as the authority layer for state transitions.

For runnable application-layer examples, see
[`context-compiler-example-integrations`](https://github.com/rlippmann/context-compiler-example-integrations).
That companion repository shows enforcement points built on compiler state,
including retrieval filtering, schema selection, tool gating, execution
authorization, gateway middleware, checkpoint continuation, and prompt
construction.

## Does it Work?

Expand All @@ -155,8 +160,6 @@ pip install context-compiler
context-compiler
```

`context-compiler` launches the interactive REPL.

Preload options keep saved rules separate from confirmation state in progress:
- `--initial-state-json` / `--initial-state-file` load saved state
(via exported state JSON).
Expand All @@ -179,9 +182,6 @@ for non-interactive usage.
context-compiler --json < input.txt
```

`--json` writes machine-readable NDJSON for non-interactive usage
(one complete JSON object per processed input line).

Preload options keep saved rules separate from confirmation state in progress:
- `--initial-state-json` / `--initial-state-file` load saved state
(via exported state JSON).
Expand Down Expand Up @@ -234,85 +234,32 @@ For normal app code, prefer the exported decision helpers (`is_clarify`,
`is_update`, `is_passthrough`, `get_clarify_prompt`, `get_decision_state`)
instead of direct key traversal.

---

### API Reference

| API | Description |
|---|---|
| `create_engine(state=None)` | Create a new compiler engine; optional `state` provides initial authoritative state (validated/canonicalized). |
| `step(user_input)` | Parse one user turn and return a deterministic `Decision`. |
| `compile_transcript(messages: Transcript)` | Replay a transcript from a fresh engine and return either final state or a confirmation prompt. |
| `engine.apply_transcript(messages: Transcript)` | Replay a transcript onto the current engine state and return either final state or a confirmation prompt. |
| `engine.state` | Read the current opaque authoritative in-memory state snapshot; for normal host reads, prefer `get_premise_value(state)` and `get_policy_items(state, ...)`. |
| `engine.has_pending_clarification()` | Return whether a confirmation-required clarification is currently pending. |
| `get_premise_value(state)` | Read the current premise value from a state snapshot. |
| `get_policy_items(state, value=None)` | Read policy items from a state snapshot (all, `use`, or `prohibit`). |
| `engine.export_json()` | Export authoritative state as JSON (`str`) for state transport/persistence. |
| `engine.import_json(payload)` | Load/restore authoritative state from exported JSON (`str`). |
| `engine.export_checkpoint()` | Export resumable checkpoint object (`Checkpoint`). |
| `engine.import_checkpoint(payload)` | Restore full checkpoint (`Checkpoint`) and return `None`. |
| `engine.export_checkpoint_json()` | Export checkpoint as canonical JSON (`str`). |
| `engine.import_checkpoint_json(payload)` | Restore checkpoint from JSON (`str`) and return `None`. |

### Controller API (Reusable Outside REPL)

These controller APIs are public package exports. You can use them directly
in app code (not just inside the REPL).

Controller quick example:

```python
from context_compiler import (
diff_has_changes,
get_step_decision,
get_step_state,
is_update,
get_preview_state_after,
create_engine,
preview,
preview_would_mutate,
state_diff,
step,
)

engine = create_engine()
See [docs/api-reference.md](docs/api-reference.md) for the full public API
reference.

before = engine.state
dry_run = preview(engine, "prohibit peanuts")
print(preview_would_mutate(dry_run)) # True
planned_change = state_diff(before, get_preview_state_after(dry_run))
print(diff_has_changes(planned_change)) # True

after_preview = engine.state
print(diff_has_changes(state_diff(before, after_preview))) # False (preview does not mutate state)

applied = step(engine, "prohibit peanuts")
print(is_update(get_step_decision(applied))) # True
print(get_step_state(applied) is not None) # True
```
Common API entry points:

| API | Description |
|---|---|
| `step(engine, user_input)` | Run one turn through the engine and return `StepResult` (`output_version`, `mode`, `decision`, `state`). |
| `preview(engine, user_input)` | Run deterministic dry-run preview and return `PreviewResult` (`output_version`, `mode`, `decision`, `state_before`, `state_after`, `diff`, `would_mutate`). Live engine state is restored after preview. |
| `state_diff(state_before, state_after)` | Return a structural `StructuralDiff` (`changed`, premise before/after, policies added/removed/changed). |
- engine lifecycle: `create_engine(...)`, `engine.step(...)`, `engine.state`,
`engine.has_pending_clarification()`
- decision helpers: `is_clarify(...)`, `is_update(...)`, `is_passthrough(...)`,
`get_clarify_prompt(...)`, `get_decision_state(...)`
- state helpers: `get_premise_value(...)`, `get_policy_items(...)`
- transcript APIs: `compile_transcript(...)`, `engine.apply_transcript(...)`
- state and checkpoint transport: `export_json(...)`, `import_json(...)`,
`export_checkpoint(...)`, `import_checkpoint(...)`
- controller APIs: `preview(...)`, `step(...)`, `state_diff(...)`

The package also exports decision-kind constants for clearer host branching:
- `DECISION_PASSTHROUGH`
- `DECISION_UPDATE`
- `DECISION_CLARIFY`
### Controller API (Reusable Outside REPL)

The package also exports decision helpers for common host-side checks:
- `is_update(decision)`
- `is_clarify(decision)`
- `is_passthrough(decision)`
- `get_clarify_prompt(decision)`
- `get_decision_state(decision)`
- `preview(engine, user_input)` performs a deterministic dry run and restores
live engine state afterward
- `step(engine, user_input)` returns a reusable result envelope around one
engine turn
- `state_diff(state_before, state_after)` summarizes structural state changes

The package exports policy value constants for explicit policy comparisons:
- `POLICY_USE`
- `POLICY_PROHIBIT`
For examples and helper accessors such as `get_step_decision(...)`,
`get_preview_state_after(...)`, `preview_would_mutate(...)`, and
`diff_has_changes(...)`, see [docs/api-reference.md](docs/api-reference.md).

---

Expand All @@ -325,8 +272,6 @@ authoritative in future turns.
- `use` = affirmative selection or preference
- `prohibit` = explicit exclusion

The compiler keeps this state in a form that your app can trust.

- Premise is a single value that can be set or replaced
- Policies are per-item (`use` or `prohibit`)
- State changes only through explicit directives
Expand Down Expand Up @@ -389,50 +334,12 @@ still belongs to the host and model workflow.
- authoritative state
- pending confirmation flow state

Checkpoint object shape:

```json
{
"checkpoint_version": 1,
"authoritative_state": {
"premise": "concise replies",
"policies": {
"docker": "use"
},
"version": 2
},
"pending": {
"kind": "replacement",
"replacement": {
"kind": "use_only",
"new_item": "kubectl",
"old_item": null
},
"prompt_to_user": "..."
}
}
```

The checkpoint shape above is an explicit serialization contract. At this
boundary, direct key access is expected.

Notes:

- `pending` is `null` when no continuation is waiting for confirmation.
- `pending` captures confirmation-required operations (for example replacement flows).
- `old_item` may be `null` for `"use_only"` when confirming “use X instead?” without an existing exact policy to replace.
- imported policy keys are normalized during `import_json` / checkpoint authoritative-state restore.
- if a policy key normalizes to `""`, the payload is invalid and is rejected.
- this keeps import-time state integrity aligned with directive-time behavior, where empty policy items are not allowed.
- checkpoint restore is full and deterministic: it restores authoritative state and pending continuation together.
- checkpoint validation is all-or-nothing; invalid payloads raise and no partial restore occurs.
- `checkpoint_version` is independent of authoritative state `version` and must be bumped when checkpoint contract shape changes (especially `pending`).

When to use checkpoint APIs:
Use state JSON when you only need authoritative state. Use checkpoint APIs when
you also need resumable continuation state across process or request
boundaries.

- stateless host or integration boundaries where engine instances are short-lived.
- resume after interruption without losing pending clarification flow.
- preserve pending confirmation flow state (`pending`) across process/request boundaries.
For the checkpoint object shape, API-level usage notes, and serialization
details, see [docs/api-reference.md](docs/api-reference.md#checkpoint-apis).

---

Expand Down Expand Up @@ -476,33 +383,35 @@ For full directive grammar and edge-case behavior, see [DirectiveGrammarSpec.md]

- [examples](examples/) — minimal usage patterns for the core authority layer
- [demos](demos/) — concrete scenarios showing how behavior differs with and without the compiler
- [`context-compiler-example-integrations`](https://github.com/rlippmann/context-compiler-example-integrations) — runnable application-layer enforcement examples built around compiler state

---

## FAQ

**Isn't this just prompt reinjection?**
No. Prompt reinjection is one way a host can use Context Compiler's
authoritative state, but Context Compiler is not a prompt-reinjection system.
It decides when state changes are allowed, when clarification is required, and
how state plus pending confirmation flow are restored. For runnable host
examples, see `context-compiler-example-integrations`.
No. Prompt construction is one downstream use of authoritative state.
Context Compiler is the authority layer that decides when state changes are
allowed, when clarification is required, and how continuation state is
restored. For runnable application-layer examples, see
[`context-compiler-example-integrations`](https://github.com/rlippmann/context-compiler-example-integrations).

**Why not just use a plain dict?**
A plain dict is enough to drive prompt construction, schema selection, and
other host behavior.
A plain dict can hold state for prompt construction, schema selection, tool
gating, and other host behavior.

Context Compiler solves a different problem: who updates that state, under which
rules, and what happens when instructions conflict.
Context Compiler solves the authority problem: who updates that state, under
which rules, and what happens when instructions conflict.

```text
User: use python_script
User: prohibit python_script
```

With a plain dict, the application must invent rules to resolve conflicts.
Context Compiler applies deterministic state-transition rules and can return
clarification instead of silently overwriting state.
Without an authority layer, the application must invent conflict-resolution and
continuation rules itself. Context Compiler applies deterministic
state-transition rules and can return clarification instead of silently
overwriting state.

---

Expand Down
27 changes: 17 additions & 10 deletions demos/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,32 @@ This demo set shows what users notice: saved authoritative state continues to
affect later turns, and where your app needs deterministic state-transition
rules.

These demos are proof-of-concept and evaluation surfaces. They isolate
authority semantics with dependency-light comparisons instead of requiring
deeper framework integration layers.

Scored demos now compare four paths:
- baseline
- reinjected-state (application-managed state text injected into the prompt, without compiler semantics)
- reinjected-state (application-managed state text injected into the prompt, used here as a simple prompt-only comparison baseline without compiler semantics)
- compiler-mediated (full transcript + saved compiler state added to the prompt)
- compiler+compact (compacted transcript + saved compiler state added to the prompt)

Runnable application-layer enforcement-point integrations live in
[`context-compiler-example-integrations`](https://github.com/rlippmann/context-compiler-example-integrations).

## Demo overview

| Demo | Behavior | Concept | Most visible with |
| :--: | --- | :--: | --- |
| [03](./03_llm_premise_guardrail.py) | Premise updates stay authoritative | fixed, repeatable premise updates | models that summarize conversation |
| [01](./01_llm_contradiction_clarify.py) | Contradiction blocking | clarification gate | small instruct models |
| [08](./08_llm_replacement_precondition.py) | Replacement precondition | invalid replacement blocked without state mutation | any model |
| [09](./09_llm_pending_clarification.py) | Pending clarification continuation | confirmation-only resolution of suspended mutation | any model |
| [06](./06_llm_context_compaction.py) | Context compaction | saved compiler state replacing transcript context | small or local models |
| [07](./07_llm_prompt_vs_state.py) | Prompt engineering comparison | prompting vs saved compiler state | any model with long transcript sensitivity |
| [02](./02_llm_constraint_guardrail.py) | Policy state stays active across turns | authoritative policy state | small or quantized models |
| [03](./03_llm_premise_guardrail.py) | Premise updates stay authoritative | fixed, repeatable premise updates | models that summarize conversation |
| [04](./04_llm_tool_denylist_guardrail.py) | Tool governance | application-layer tool gating from saved state | general assistant models |
| [05](./05_llm_prompt_drift_vs_state.py) | Prompt drift | long transcript failure | weaker long-context models ([see Demo 5 note](#demo-5-stress-ladder-turns)) |
| [06](./06_llm_context_compaction.py) | Context compaction | saved compiler state replacing transcript context | small or local models |
| [07](./07_llm_prompt_vs_state.py) | Prompt engineering comparison | prompting vs saved compiler state | any model with long transcript sensitivity |
| [08](./08_llm_replacement_precondition.py) | Replacement precondition | invalid replacement blocked without state mutation | any model |
| [09](./09_llm_pending_clarification.py) | Pending clarification continuation | confirmation-only resolution of suspended mutation | any model |

Stronger frontier models may show these behaviors less often, but the same
patterns still appear in real applications.
Expand Down Expand Up @@ -158,13 +165,13 @@ Notes:
- There are **8 scored demos** (`01`–`05`, `07`, `08`, `09`). `06_context_compaction` is informational and excluded from PASS/FAIL totals.
- Anthropic runs in this repo are executed through the `openai_compatible` provider path.
- `PASS` means the demo-specific expected-behavior check for that path succeeded; `FAIL` means it did not.
- `reinjected-state` can be enough for some persistence cases; this comparison shows where app-side state rules add value.
- Scored checks focus on app-side state rules (for example blocked mutation and confirmation-only resolution), not model prose quality. `reinjected-state` remains plain text injection only.
- `reinjected-state` can be enough for some persistence cases; in this demo set it is intentionally used as a prompt-only comparison baseline.
- Scored checks focus on app-side authority rules (for example blocked mutation and confirmation-only resolution), not model prose quality. `reinjected-state` remains plain text injection only.
- Interpretation:
- Demos `01`-`05` and `07` mostly test persistence and policy-following behavior across turns.
- Demos `08`/`09` test rules for when state is allowed to change.
- Demos `08` and `09` cover cases prompt text does not implement by itself, such as checking whether replacement is allowed and waiting for confirmation before saving changes.
- Plain prompt reinjection can produce reasonable answers, but it does not run these checks by itself.
- Demos `08` and `09` cover authority semantics prompt text does not implement by itself, such as replacement preconditions, blocked mutations, and waiting for confirmation before saving changes.
- Plain prompt reinjection can produce reasonable answers, but it does not run these authority checks by itself and is not the only or preferred production integration pattern.
- Similar outcomes across models in `08`/`09` reflect app behavior limits, not model leaderboard ranking.

### Demo 05 example (prompt drift under longer context)
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
## Core Concepts
- [Architecture boundaries](architecture.md)
- [Directive Grammar (exact command forms the engine accepts)](DirectiveGrammarSpec.md)
- [Public API reference](api-reference.md)

## Evaluation & Evidence
- [Demo results matrix](demos-results.md)
Expand Down
Loading
Loading