chore: prepare 0.7.1 docs and version bump

README.md

@@ -1,27 +1,56 @@
-
 # Context Compiler
 
 [![PyPI version](https://img.shields.io/pypi/v/context-compiler)](https://pypi.org/project/context-compiler/)
 [![Python versions](https://img.shields.io/pypi/pyversions/context-compiler)](https://pypi.org/project/context-compiler/)
 [![License](https://img.shields.io/pypi/l/context-compiler)](https://pypi.org/project/context-compiler/)
 
-Context Compiler lets users set rules and corrections that actually stick.
-It helps applications keep explicit user instructions consistent across turns.
-It stores premise and policy rules outside the model, so corrections do not drift or conflict over time.
+Some behaviors require explicit host-side state machinery.
+
+Context Compiler is a deterministic host-side state layer for LLM applications.
+It handles explicit state transitions for premise and policies so that mutation
+rules are fixed and repeatable.
+
+## What prompting and reinjection can do
+
+Prompting and reinjection are useful. In many real systems, reinjecting saved
+state text is enough to keep instructions and policies persistent across turns.
+
+Context Compiler adds host-owned transition rules for behaviors that plain text
+reinjection does not implement by itself: replace `X` only if `X` exists, block
+conflicting changes and ask for confirmation, and restore saved state plus
+pending confirmations from checkpoints.
+
+## What prompting cannot do by itself
+
+Prompt text (including reinjected state text) helps, but it does not give your
+app controlled rules for when state can change. By itself, it does not provide:
+
+- rules your app controls for state changes
+- replacement precondition checks (`use X instead of Y` when `Y` may be absent)
+- confirmation flows that must complete before anything else changes
+- clear decisions about when a change is blocked
+- reliable checkpoint restore for both saved state and pending confirmation flow
 
-LLMs are good at conversation, but bad at consistently following long-term rules and corrections. Constraints drift, corrections conflict, and long chats can become inconsistent.
+## What Context Compiler provides
 
-The model writes responses. The compiler stores premise and policy rules.
+Context Compiler provides fixed host-side state machinery:
 
-Context Compiler is a deterministic control layer for LLM applications. It processes explicit user instructions before model calls so applications can reliably enforce premise and policy constraints.
+- deterministic directive handling for explicit user state changes
+- clarification instead of silent overwrite for blocked/ambiguous changes
+- pending confirmation flows that must resolve before anything else changes
+- checkpoint export/import for restoring saved state and pending confirmation flow
+- structured saved state that the host can pass to the model
 
-Context Compiler handles the cases that become operationally complicated quickly:
+The model generates responses. The compiler owns state transitions.
 
-- replacing a policy that may or may not exist yet
-- detecting contradictory directives before applying them
-- enforcing valid state transitions (you can't change what isn't set)
-- persisting authoritative state across stateless API boundaries
-- resuming after an interrupted clarification flow
+## How the compiler metaphor works
+
+Context Compiler treats important instructions as structured state instead of
+temporary prompt text.
+
+Like a compiler, it parses input, validates it, applies fixed rules, and
+produces a stable representation the host can use. It is not source-code
+compilation and not a reasoning model.
 
 ## Does it work?
 
@@ -32,12 +61,12 @@ Yes, on the current scored demo set.
 - Across tested models, compiler-mediated paths pass all scored scenarios; baseline behavior is model-dependent.
 
 Interpretation guide:
-- Persistence/policy-following demos test whether instructions keep applying across turns.
-- State-transition demos (`08`/`09`) test whether the host enforces state changes in a fixed, repeatable way.
-- Demos `08`/`09` are capability checks, not general model-quality rankings.
-- In those demos, baseline or reinjected-state can sound reasonable and still `FAIL` because required host-side transition checks are not present.
+- Demos `01`-`05` and `07` focus on persistence and policy-following behavior.
+- Demos `08`/`09` focus on rules for when state is allowed to change.
+- Demos `08`/`09` show what prompt text does not implement by itself.
+- Plain reinjection can produce plausible responses, but it does not check whether replacement is allowed or wait for confirmation before saving changes.
 
-→ [Full results and demo output](demos/README.md)  
+→ [Full results and demo output](demos/README.md)
 Canonical matrix: [docs/demos-results.md](docs/demos-results.md)
 
 ## Quickstart
@@ -52,9 +81,9 @@ context-compiler --json < input.txt
 `context-compiler` launches the interactive REPL.
 
 `--with-preprocessor` enables the experimental preprocessor before each REPL turn
-(heuristic + validation only). Near-miss inputs are not rewritten and are
-passed through to the engine, which continues to return clarify behavior for
-those forms.
+(simple rule-based checks plus conservative validation). For near-miss inputs,
+the preprocessor does not rewrite the text. It passes the input to the engine,
+and the engine can return `clarify`.
 
 `--json` enables machine-readable NDJSON output for non-interactive usage
 (one complete JSON object per processed input line).
@@ -66,7 +95,7 @@ Preload options keep saved rules separate from in-progress confirmation state:
   continuation checkpoint (saved state + pending confirmation state).
 
 REPL commands (controller layer, not engine directives):
-- `state` shows current authoritative state.
+- `state` shows current saved state.
 - `preview <input>` runs deterministic dry-run without mutating live state.
 - `step <input>` is an explicit alias of normal bare-input step behavior.
 
@@ -117,36 +146,25 @@ uv run pytest
 
 ---
 
-## Why “Compiler”?
-
-Context Compiler treats explicit user directives as inputs to a fixed, repeatable process.
-
-Instead of relying on the LLM to remember constraints across a conversation, user instructions are compiled into structured state before the model runs.
-
-The idea is similar to a traditional compiler: user directives are translated into a structured representation that the rest of the system can rely on.
-
----
-
 ## FAQ
 
 **Is this just prompt reinjection?**
 Reinjection helps with persistence, and it remains useful. Context Compiler
-addresses a different boundary: authoritative state transitions. It defines how
-state mutates before generation, with deterministic externalized semantics.
+handles a different problem: rules for when state is allowed to change.
 
 Examples:
 - replacement semantics (`use X instead of Y`) when `Y` may not exist
 - contradiction detection before applying a mutation
 - lifecycle enforcement (for example, you cannot change an unset premise)
 - pending clarification flows that must be resolved before other mutations
 
-In short: reinjection carries state forward; Context Compiler governs how that
-state is created and changed.
+In short: reinjection carries state forward; Context Compiler decides when your
+app should change state.
 
 **Isn’t this just prompt engineering?**
 It complements prompt engineering, but solves a different problem. Prompting
-shapes model behavior; Context Compiler provides a deterministic state layer
-that updates only through explicit directives.
+shapes model behavior. Context Compiler enforces state rules and updates state
+only through explicit directives.
 
 ---
 
@@ -166,36 +184,38 @@ Later in the conversation:
 User: how should I make this curry?
 ```
 
-Your app sends the saved state to the model so the rule still applies on later turns.
+Your host sends the saved policy state with this later request, so the model is
+constrained by explicit state (`peanuts: prohibit`) instead of relying on memory
+of earlier conversation text.
 
 ---
 
 ## Deterministic behavior (examples)
 
-Context Compiler externalizes mutation semantics so transitions are deterministic.
+Context Compiler makes mutation rules explicit so behavior stays repeatable.
 
 **Explicit directive**
 ```text
 set premise concise replies
 ```
 - Base model: silently accepts / rewrites
-- Context Compiler: applies a deterministic state update
+- Context Compiler: applies a repeatable state update
 
 **State-dependent operation**
 ```text
 clear state
 use podman instead of docker
 ```
 - Without explicit state transition rules: behavior depends on host/model handling
-- Context Compiler: returns deterministic clarify behavior before mutation
+- Context Compiler: returns `clarify` before changing state
 
 **Lifecycle enforcement**
 ```text
 clear state
 change premise to formal tone
 ```
 - Without explicit transition checks: behavior depends on host/model handling
-- Context Compiler: clarifies and preserves authoritative state
+- Context Compiler: asks for clarification and keeps saved state unchanged
 
 ---
 
@@ -302,7 +322,7 @@ The internal structure of the state is intentionally opaque to host applications
 - `export_json()` / `import_json()` transport **authoritative state only**
 - checkpoint APIs transport **serialized continuation**:
   - authoritative state
-  - pending confirmation-required continuation state
+  - pending confirmation flow state
 
 Checkpoint object shape:
 
@@ -344,7 +364,7 @@ When to use checkpoint APIs:
 
 - stateless host/integration boundaries where engine instances are short-lived.
 - resume after interruption without losing pending clarification flow.
-- preserve confirmation-required continuation state (`pending`) across process/request boundaries.
+- preserve pending confirmation flow state (`pending`) across process/request boundaries.
 
 ---
 

demos/README.md

@@ -1,13 +1,13 @@
 # LLM Demos
 
-These scripts show common ways LLM conversations can go wrong.
+These scripts show common reliability limits in LLM app behavior.
 
 They compare normal prompting with an approach where the application tracks
 important instructions explicitly instead of relying only on the conversation
 history. The scripts are designed to produce consistent results so the
 behavior is easy to see.
-This demo set shows what users notice: rules and corrections keep applying
-later in the conversation instead of fading over time.
+This demo set shows what users notice: rules and corrections can keep applying
+later in the conversation, and where your app needs explicit state rules.
 
 Scored demos now compare four paths:
 - baseline
@@ -147,10 +147,14 @@ Notes:
 - There are **8 scored demos** (`01`–`05`, `07`, `08`, `09`). `06_context_compaction` is informational and excluded from PASS/FAIL totals.
 - Anthropic runs in this repo are executed through the `openai_compatible` provider path.
 - `PASS` means the demo-specific expected-behavior check for that path succeeded; `FAIL` means it did not.
-- `reinjected-state` can be enough for some persistence cases; this comparison is intended to show where deterministic state semantics add value.
-- Anti-overfitting guardrail: demos score host/state-transition invariants (for example blocked mutation, pending continuation, confirmation-only resolution), not model prose quality. Reinjected-state remains plain text injection only and does not include hidden compiler semantics.
-- Interpretation: demos `01`-`05` and `07` focus on persistence and policy-following across turns, while demos `08`/`09` focus on fixed host-side state-transition rules.
-- For demos `08`/`09`, similar outcomes across models are about architecture limits, not provider/model leaderboard ranking.
+- `reinjected-state` can be enough for some persistence cases; this comparison shows where app-side state rules add value.
+- Scored checks focus on app-side state rules (for example blocked mutation and confirmation-only resolution), not model prose quality. `reinjected-state` remains plain text injection only.
+- Interpretation:
+- Demos `01`-`05` and `07` mostly test persistence and policy-following behavior across turns.
+- Demos `08`/`09` test rules for when state is allowed to change.
+- Demos `08` and `09` cover cases prompt text does not implement by itself, such as checking whether replacement is allowed and waiting for confirmation before saving changes.
+- Plain prompt reinjection can produce reasonable answers, but it does not run these checks by itself.
+- Similar outcomes across models in `08`/`09` reflect app behavior limits, not model leaderboard ranking.
 
 ### Demo 05 example (prompt drift under longer context)
 

docs/demos-results.md

@@ -9,8 +9,8 @@ Note: this published matrix predates the `reinjected-state` path added in the 0.
 - Scored demos: `01`, `02`, `03`, `04`, `05`, `07` (6 total)
 - Informational demo: `06_context_compaction` (excluded from PASS/FAIL totals)
 
-Methodology note (2026-05): the demo suite now also includes scored state-transition
-semantics demos `08` and `09`. The published matrix below predates those additions and
+Methodology note (2026-05): the demo suite now also includes scored demos `08`
+and `09` for rules about when state is allowed to change. The published matrix below predates those additions and
 has not yet been fully rerun with the expanded scored set.
 
 ## Results Matrix
@@ -67,10 +67,10 @@ Scoring behavior uses post-audit oracle/checker logic in demos and shared helper
 - `demos/09_llm_pending_clarification.py`
 - shared parsing/helpers in `demos/common.py`
 
-Anti-overfitting note: scored checks are designed around deterministic host/state
-invariants rather than preferred wording in model outputs. The `reinjected-state`
-path is intentionally limited to plain state text injection and does not include
-compiler precondition or pending-state semantics.
+Scored checks focus on app-side state rules rather than preferred wording in
+model outputs. The `reinjected-state` path is plain state text injection and
+does not include compiler checks like replacement preconditions or pending
+confirmation handling.
 
 ### Run metadata
 
@@ -83,10 +83,12 @@ compiler precondition or pending-state semantics.
 
 - Live demo runs are **evidence/smoke tests** across real model/provider behavior.
 - Deterministic test suites (unit/property tests) are the **regression authority** for oracle and engine contracts.
-- Persistence demos and transition-semantics demos should be interpreted differently:
-- `01`-`05`, `07`: persistence and policy-following under transcript pressure.
-- `08`/`09`: host-side transition checks (for example, replacement preconditions and pending-confirmation handling).
-- Demos `08`/`09` are not general LLM quality benchmarks. Baseline and reinjected-state can produce plausible text and still `FAIL` when those host-side transition checks are missing.
+- Persistence demos and state-change-rule demos should be interpreted differently.
+- Demos `01`-`05` and `07` mostly test persistence and policy-following under transcript pressure.
+- Demos `08`/`09` test rules for when state is allowed to change.
+- Demos `08` and `09` cover cases prompt text does not implement by itself, such as checking whether replacement is allowed and waiting for confirmation before saving changes.
+- Plain prompt reinjection can produce reasonable answers, but it does not run these checks by itself.
+- Demos `08`/`09` are not general LLM quality benchmarks. Baseline and reinjected-state can produce plausible text and still `FAIL` when those app-side checks are missing.
 
 ## Demo 05 Long-Transcript Stress (Exploratory Frontier Runs)
 
@@ -99,15 +101,15 @@ records the standard scored demo configuration.
 - In those exploratory runs, `reinjected-state`, `compiler`, and `compiler+compact`
   continued to preserve premise-consistent behavior.
 
-This is exploratory evidence rather than deterministic benchmark authority. Reinjection can
-be sufficient in some persistence scenarios, while compiler-mediated paths still provide
-deterministic state-transition semantics.
+This is exploratory evidence, not benchmark authority. Reinjection can be
+enough in some persistence scenarios, while compiler-mediated paths still
+provide explicit state-change rules.
 
 ## Local Ollama Context-Size Sweep (0.7.1 Experiment)
 
 This section reports the refreshed local-only matrix with the `reinjected-state`
-path and explicit context-size ladder runs. Historical hosted-provider matrix rows
-above are preserved as originally recorded.
+path and explicit context-size ladder runs. Historical hosted-provider matrix
+rows above remain as originally recorded.
 
 ### Commands Run