Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Contributing

Thanks for your interest in improving this project. Contributions are welcome.

## Workflow

Contributions are typically submitted via fork and pull request:

- fork the repository
- create a feature branch
- run tests and pre-commit checks
- open a pull request

## Development Setup

```bash
uv sync --group dev
```

## Running Tests

```bash
uv run pytest
```

## Code Quality

```bash
pre-commit run --all-files
```

## Scope of Changes

- keep pull requests focused
- include tests if behavior changes
- open an issue first for large design changes
220 changes: 67 additions & 153 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,30 @@

# Context Compiler

Deterministic conversational state for LLM systems.
[![PyPI version](https://img.shields.io/pypi/v/context-compiler)](https://pypi.org/project/context-compiler/)
[![Python versions](https://img.shields.io/pypi/pyversions/context-compiler)](https://pypi.org/project/context-compiler/)
[![License](https://img.shields.io/pypi/l/context-compiler)](https://pypi.org/project/context-compiler/)

Modern language models are strong at reasoning but unreliable at maintaining consistent state across interactions. Corrections compete with earlier statements, constraints disappear, and long conversations accumulate contradictions.
A deterministic directive engine that converts explicit user instructions
into structured conversational state for LLM applications.

Modern language models reason well but are unreliable at maintaining
consistent state across interactions.

Corrections compete with earlier statements, constraints disappear,
and long conversations accumulate contradictions.

The **Context Compiler** introduces a deterministic state layer that governs authoritative conversational state independently of the model.

The model performs reasoning and generation.
The compiler manages facts and constraints.
The model performs reasoning and generation while the compiler manages facts and constraints. Once accepted, directives remain authoritative until explicitly corrected or reset.

## Installation

Once a directive is accepted, it becomes authoritative for the remainder of the session.
- Python 3.11+
- `pip install context-compiler`
- Dev/test: `uv sync --group dev` and `uv run pytest`
- Examples: see [examples/README.md](examples/README.md)
- Demonstrations: see [demos/README.md](demos/README.md)

---

Expand Down Expand Up @@ -46,22 +60,6 @@ The host supplies the authoritative state to the model so the constraint persist

---

## Why Not Just Prompt Engineering?

Prompt instructions are soft and easy to lose across long interactions.

The Context Compiler gives the host an **authoritative state representation** that is independent of transcript drift.

Only **explicit user directives** can modify state.

In other words:

- the model reasons
- the compiler decides whether state changes
- the host controls when the LLM runs

---

## Architecture

```text
Expand All @@ -76,30 +74,8 @@ Host Application
LLM
```

The compiler governs conversational state. It never calls the LLM.

The LLM performs reasoning and generation but cannot modify authoritative state.

The host application decides whether the model runs based on the compiler’s `Decision`.

### Compiler responsibilities

The compiler:

1. Parses user input
2. Detects explicit directives
3. Ensures mutations are unambiguous
4. Returns a deterministic `Decision`

The compiler **never calls the LLM**.

### Host responsibilities

The host:

- displays clarification prompts
- calls the LLM when allowed
- formats prompts using compiled state
The compiler governs authoritative conversational state and never calls the LLM.
The host decides whether to call the model based on the returned `Decision`.

---

Expand All @@ -122,6 +98,30 @@ Meaning:
| update | forward input with updated state |
| clarify | show `prompt_to_user` and do not call the LLM |

### Host Integration Example

```python
engine = create_engine()
decision = engine.step(user_input)

if decision["kind"] == "clarify":
show_to_user(decision["prompt_to_user"])
else:
state = decision["state"] or engine.state
messages = build_messages(state, user_input)
render(call_llm(messages))
```

### API Reference

| API | Description |
|---|---|
| `create_engine(...)` | Create a new compiler engine, optionally with replacement initial state. |
| `step(user_input)` | Parse one user turn and return a deterministic `Decision`. |
| `engine.state` | Read or replace full in-memory authoritative state. |
| `export_json()` | Export current state as JSON for persistence/transport. |
| `import_json(payload)` | Load state from exported JSON payload. |

---

## State Model
Expand All @@ -142,57 +142,22 @@ The compiler maintains an authoritative state:

## State Access and Persistence

The compiler exposes authoritative state through in-memory replacement APIs
and JSON persistence/transport APIs.

Supported host interaction model:

```python
engine = create_engine()
engine = create_engine(state=initial_state_obj)
decision = engine.step(user_input)
state = engine.state
engine.state = replacement_state_obj
payload = engine.export_json()
engine.import_json(payload)
```

API boundaries:

- `step()` performs directive-driven mutation.
- `engine.state` / `engine.state = ...` are in-memory inspection/replacement APIs.
- `export_json()` / `import_json()` are persistence/transport APIs.
- No imperative convenience mutation methods are provided; operations such as
`reset policies` and `clear state` are handled through directive input to `step()`.

Example persistence lifecycle:

```python
payload = engine.export_json()
save_to_storage(payload)

restored_payload = load_from_storage()
engine = create_engine()
engine.import_json(restored_payload)
```
Hosts may inspect or replace in-memory state (`engine.state`) or persist it using `export_json()` and `import_json()`. State changes occur only through directives processed by `step()`. Storage is managed by the host application.

The compiler does not manage storage or snapshots. Persistence policies
belong to the host application.
### Fact Schema

**Note**
The fact schema currently contains a single exclusive slot: `facts["focus.primary"]`.

This slot exists to demonstrate deterministic fact replacement and correction semantics.
Richer fact schemas may be introduced in future releases.
The current schema contains a single exclusive slot: `facts["focus.primary"]`.
This demonstrates deterministic fact replacement and correction behavior.
Richer schemas may be introduced in future releases.

### State Properties

- **Facts are exclusive** (last write wins)
- **Policies are additive**
- **No inference or semantic reasoning**
- **State is deterministic**
- Facts are exclusive (last write wins)
- Policies are additive
- No inference or semantic reasoning

The same input sequence always produces the same state.
Identical input sequences always produce identical compiler state.
LLM responses may still vary unless deterministic decoding is used by the host.

---

Expand Down Expand Up @@ -295,22 +260,9 @@ After `clear state`:

## Examples

The `examples/` directory contains small integration demonstrations.

- [Persistent guardrails](examples/01_persistent_guardrails.py)
Demonstrates constraints persisting across turns.

- [Configuration with correction](examples/02_configuration_and_correction.py)
Shows deterministic fact replacement.
Integration examples are available in the [examples/](examples/) directory.

- [Ambiguity detection with clarification](examples/03_ambiguity_with_clarification.py)
Demonstrates clarification before state mutation.

- [Tool governance for agents](examples/04_tool_governance_denylist.py)
Shows how host applications can block tools using compiler policies.

- [LLM integration pattern](examples/05_llm_integration_pattern.py)
Demonstrates the host control flow around the `Decision` API.
See [examples/README.md](examples/README.md) for walkthroughs.

---

Expand All @@ -331,66 +283,28 @@ python examples/01_persistent_guardrails.py
Run tests:

```bash
pytest
uv run pytest
```

---

## Guarantees

The compiler enforces several invariants:

- State never changes without explicit directive or confirmation
- The same input sequence always produces the same state
- LLM output never affects state
- No mutation occurs during clarification
- Administrative state replacement clears pending clarification state
- Facts are exclusive
- Policies are additive
- Pending clarification blocks mutation

---

## Implementation Status
- State changes only through explicit user directives or confirmation.
- Identical input sequences produce identical compiler state.
- Model responses never modify compiler state.
- Ambiguous directives trigger clarification instead of changing state.

The current implementation provides the deterministic directive compiler
defined in [this document](/docs/M1Design.md).
These invariants are verified through behavioral tests and Hypothesis-based property tests.

---

## Future Work

This project intentionally focuses on the deterministic directive compiler itself.

Higher-level systems such as session scope management, memory layers, agent tooling,
or context routing belong in host applications or separate projects built on top of
the compiler.

---

## Design Philosophy

The Context Compiler deliberately avoids:
## Design Notes

- semantic reasoning
- ontology inference
- machine-learning parsing
- transcript reconstruction
- passive memory

Authoritative state must originate only from **explicit user directives**.

---

## Specification

The authoritative behavioral specification is:

```text
M1Design.md
```
More detailed design and milestone documents are available in:

Future milestone documents describe potential extensions but are not normative for current behavior.
- [Project overview](docs/DescriptionAndMilestones.md)
- [M1 design document](docs/M1Design.md)

---

Expand Down
6 changes: 6 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Documentation

This directory contains the project design and milestone specifications.

- `DescriptionAndMilestones.md` — project overview and milestone roadmap
- `M1Design.md` — authoritative behavioral specification for the deterministic directive engine
28 changes: 28 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Examples

This directory contains small integration examples showing typical host-side usage of the Context Compiler.

## 01_persistent_guardrails.py

Demonstrates how a prohibition persists as authoritative state across later turns.
Shows the host using compiled state to keep constraints active.

## 02_configuration_and_correction.py

Demonstrates deterministic fact replacement with explicit correction.
Shows last-write-wins behavior for `facts.focus.primary`.

## 03_ambiguity_with_clarification.py

Demonstrates ambiguity detection before state mutation.
Shows how the host handles `Decision.kind == "clarify"` and resumes after confirmation.

## 04_tool_governance_denylist.py

Demonstrates tool-governance policy handling via prohibition directives.
Shows how hosts can prevent denied tools from being selected.

## 05_llm_integration_pattern.py

Demonstrates the end-to-end host control flow around `Decision`.
Shows when to clarify, when to call the model, and how to include compiled state in prompts.
21 changes: 20 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,32 @@ build-backend = "hatchling.build"

[project]
name = "context-compiler"
version = "0.1.0"
version = "0.2.0"
description = "Deterministic conversational state engine for LLM applications."
readme = "README.md"
requires-python = ">=3.11"
license = { text = "Apache-2.0" }
authors = [{ name = "Robert Lippmann" }]
dependencies = []
keywords = [
"llm",
"conversation-state",
"state-machine",
"ai-infrastructure",
]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3",
"Topic :: Software Development :: Libraries",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]

[project.urls]
Homepage = "https://github.com/rlippmann/context-compiler"
Repository = "https://github.com/rlippmann/context-compiler"
Issues = "https://github.com/rlippmann/context-compiler/issues"

[project.scripts]
context-compiler = "context_compiler.repl:main"
Expand Down
Loading
Loading