From 025c3fba4477eee036555f86ebd711edcd62b9dd Mon Sep 17 00:00:00 2001 From: Robert Lippmann Date: Mon, 23 Mar 2026 00:59:58 -0400 Subject: [PATCH] docs: align milestone language with 0.5 premise/policy model --- docs/DescriptionAndMilestones.md | 23 +- docs/DirectiveGrammarSpec.md | 472 +++++++++++++------------------ 2 files changed, 203 insertions(+), 292 deletions(-) diff --git a/docs/DescriptionAndMilestones.md b/docs/DescriptionAndMilestones.md index 5bfc6a7..0ada52e 100644 --- a/docs/DescriptionAndMilestones.md +++ b/docs/DescriptionAndMilestones.md @@ -11,7 +11,7 @@ conversations, and long-term state accumulates contradictions rather than resolving them. This project introduces a deterministic state layer that governs authoritative state independently of the model. The model performs -interpretation and generation, while the state engine manages deterministic conversational state (facts and policies). By separating reasoning +interpretation and generation, while the state engine manages deterministic authoritative state (premise and policies). By separating reasoning from state authority, the system improves reliability without requiring model retraining. The system never derives authoritative state from model responses; only user directives modify state. @@ -28,24 +28,23 @@ The state engine is authoritative and model-independent. Model output is never interpreted to derive or modify state. All state transitions originate from explicit user directives. +Behavioral details are authoritative in `docs/DirectiveGrammarSpec.md`. + ## Project Milestones ### M1 — Deterministic State Engine (implemented) **Goal** Explicit user commitments persist reliably within a conversation. -A correction means replacing a previously set fact, not evaluating conversational accuracy. - -In M1 the fact schema intentionally contains a single exclusive slot (`facts["focus.primary"]`). -This slot demonstrates deterministic fact replacement and correction semantics. +A change directive means replacing previously set authoritative state, not evaluating conversational accuracy. -Policies (`policies.prohibit`) provide the primary mechanism for persistent conversational constraints. -Richer fact schemas may be introduced in future milestones. +M1 established deterministic state transitions and explicit clarification behavior. +The current authoritative state shape and directive semantics are defined in `DirectiveGrammarSpec.md` (0.5 / schema version 2). **Core capability:** -- Recognize high-confidence user directives (facts and policies) -- Apply corrections as deterministic replacements +- Recognize explicit user directives that mutate premise or policies +- Apply explicit state changes as deterministic replacements - Block ambiguous updates until clarified - Maintain an authoritative state independent of prior messages - Provide structured state for host-provided model context @@ -53,12 +52,12 @@ Richer fact schemas may be introduced in future milestones. **Deliverables:** - Directive grammar (conservative pattern set) -- State data model (facts + policies) -- Deterministic update rules (exclusive vs additive slots) +- State data model (authoritative conversational state) +- Deterministic update rules for explicit directives and clarification - Clarification mechanism for ambiguous mutations - Context serialization interface (`export_json` / `import_json`, state → host application) - Reference integration harness (example host) -- Tests: persistence and non-regression of corrections +- Tests: persistence and non-regression of deterministic state updates **User-visible outcome:** diff --git a/docs/DirectiveGrammarSpec.md b/docs/DirectiveGrammarSpec.md index f88e6bd..3a42163 100644 --- a/docs/DirectiveGrammarSpec.md +++ b/docs/DirectiveGrammarSpec.md @@ -1,376 +1,288 @@ -# Context Compiler — Directive Grammar Specification +# Context Compiler - Directive Grammar Specification (0.5) ## Goal -Ensure explicit user corrections and constraints persist reliably within -a single conversation. -This specification provides authoritative conversational state. It does **not** perform -reasoning, inference, entity resolution, cross-session memory, or -planning. +Provide deterministic, explicit, and model-independent conversational state updates. -### 1. Terminology +This specification defines the authoritative state machine for directive handling. +It does not perform reasoning, inference, entity modeling, natural-language understanding, +or interpretation of assistant output. -| | | -|------------|----------------------------------------------------------------| -| Term | Meaning | -| User Input | Raw text from the user | -| Directive | A statement attempting to change authoritative state | -| Policy | A standing constraint (“don’t do X”) | -| Fact | A configuration choice about the current focus (“I’m using X”) | -| State | Current authoritative truth snapshot | -| Pending | Awaiting clarification before mutation | -| Decision | Compiler instruction returned to host | +## 1. Terminology -### 2. System Responsibilities +| Term | Meaning | +|---|---| +| User Input | Raw text from the user | +| Directive | A string that matches one of the explicit grammar productions in Section 5 | +| Premise | Single sticky explicit slot controlled only by premise directives | +| Policy | Per-item authoritative state: `"use"` or `"prohibit"` | +| State | Current authoritative snapshot | +| Pending Clarification | A blocked mutation awaiting explicit user confirmation | +| Decision | Compiler instruction returned to host | -Compiler Responsibilities +## 2. System Responsibilities The compiler: -1. Parses user input -2. Determines if state changes -3. Ensures mutations are unambiguous +1. Parses user input against explicit grammar +2. Applies deterministic state transitions only when valid +3. Rejects contradictory or invalid mutations with `clarify` 4. Returns a deterministic `Decision` -The compiler never calls the LLM. -Mutations are expressed through directive input to `step()`, including reset/clear operations. -Imperative convenience methods such as `reset_policies()` and `clear_state()` are not part of the public API. +The compiler never calls an LLM. +All authoritative mutations originate from user directives passed to `step()`. -### 3. Host Responsibilities +## 3. Host Responsibilities The host: - Displays clarification prompts -- Calls the LLM when allowed -- Formats prompts using provided state -- May read state snapshots directly, but should prefer public helper accessors where available. +- Calls the LLM only when `Decision.kind` allows it +- Formats model context from compiler state -Current helpers: -- `get_focus_value(state)` -- `get_prohibited_items(state)` +## 4. Decision API Contract -These helpers are read-only conveniences for state snapshots to reduce direct -coupling to nested layout. They do not modify compiler state and are not -semantic/compiler primitives. +```python +class Decision(TypedDict): + kind: Literal["passthrough", "update", "clarify"] + state: dict | None + prompt_to_user: str | None +``` -### 4. Decision API Contract +Semantics: -`python - class Decision(TypedDict): - kind: Literal["passthrough", "update", "clarify"] - state: dict | None - prompt_to_user: str | None -` - -#### Semantics - -| | | -|-------------|---------------------------------------| -| kind | Host behavior | -| passthrough | forward user input to LLM | -| update | forward user input with updated state | -| clarify | DO NOT call LLM; show prompt_to_user | +- `passthrough`: forward user input to LLM +- `update`: forward user input with updated compiler state +- `clarify`: do not call LLM; display `prompt_to_user` The compiler always returns a `Decision`. -### 4. State Model +## 5. State Model State is a deterministic snapshot: -`json - { - "facts": { - "focus.primary": null - }, - "policies": { - "prohibit": [] - }, - "version": 1 - } -` - -#### Properties - -- Keys are explicit -- Values are opaque strings -- Facts are exclusive (last write wins) -- Policies are additive sets -- No reasoning or inference occurs - -### 5. Directive Parsing Rules - -The compiler mutates state only on high-confidence directives. - -#### 5.1 Hard Negative Directives - -Accepted patterns: - -- "don't X" -- "do not X" -- "never X" -- "please don't X" - -Produces: - - POLICY_ADD(normalize(X)) - -For hard-negative directives, `normalize(X)` is applied to each policy -payload item before storage. - -Soft preference phrases like `"avoid X"` and `"refrain from X"` are -not hard directives in M1. They are treated as passthrough input and do -not mutate authoritative state. - -#### 5.2 Hard Positive Directives -Accepted patterns: +```json +{ + "premise": null, + "policies": {}, + "version": 2 +} +``` -- "use X" -- "set X" -- "I am using X" -- "I'm using X" +Where: -Polite prefixes such as `"please"` may be tolerated and ignored by the parser. -For example, `please use X` is treated the same as `use X`. +- `premise`: `string | null` +- `policies`: `dict[string, "use" | "prohibit"]` +- `version`: integer schema version. The 0.5 design maps to schema version `2`. +- Policy key absence means no policy for that item. -Produces: +Properties: - FACT_SET(key="focus.primary", value=X) +- Premise is explicit and sticky. +- Policies are authoritative per item. +- No policy ordering. +- No policy recency semantics. +- No policy history semantics. -“Using” statements set the current discussion focus, not inventory. -Hard-positive directives store fact values as opaque strings and do not -apply `normalize(X)` to the stored fact value. +## 6. Normalization -#### 5.3 Correction Markers +`normalize_item(X)` for policy item keys: -Accepted markers: - -- "actually" -- "I meant" -- "correction:" -- "no," +1. Unicode NFKC normalization +2. Lowercase +3. Collapse internal whitespace to single spaces +4. Remove leading articles: `a`, `an`, `the` +5. Normalize apostrophes (`dont` -> `don't`) -Corrections apply only to the most recently updated exclusive -fact. If no such fact exists → clarification required. -Correction payloads must represent a fact replacement value. If a -correction payload appears to invoke another directive family (for -example hard-negative directives, allow/removal directives, or -reset/clear commands) → clarification required and no state mutation. +Premise values are stored as opaque strings with minimal sanitation only: -Example: +1. Unicode normalization +2. Apostrophe normalization +3. Whitespace collapse - use Nord Stage 4 - actually don't use docker +No stemming, synonym mapping, ontology, or semantic interpretation is allowed. -→ `Decision.kind = "clarify"` and state remains unchanged. +## 7. Directive Grammar (Explicit Only) -#### 5.4 Allow / Removal Directives +Only the exact productions below are directives. +All other input is `passthrough` unless Section 8 says `clarify`. -Accepted patterns: +```text +SET_PREMISE := "set premise " VALUE +CHANGE_PREMISE := "change premise to " VALUE +USE_ITEM := "use " ITEM +PROHIBIT_ITEM := "don't use " ITEM +REPLACE_USE := "use " ITEM " instead of " ITEM +CLEAR_PREMISE := "clear premise" +RESET_POLICIES := "reset policies" +CLEAR_STATE := "clear state" +``` -- "X is fine" -- "allow X" -- "you can X" +Notes: -Produces: +- `VALUE` and `ITEM` are non-empty raw substrings after their prefix. +- `ITEM` is normalized via `normalize_item` before policy lookup/storage. +- `VALUE` is stored using premise sanitation from Section 6. +- No conversational aliases are directives (for example: `actually`, `I meant`, `allow`, `you can`, `set X`, `I'm using X`). - POLICY_REMOVE(normalize(X)) +## 8. State Transition Semantics -If X not present → no-op. -For allow/removal directives, `normalize(X)` is applied to each policy -payload item used for removal matching. +### 8.1 Premise lifecycle -#### 5.5 List Handling +- `set premise X`: + - valid only if `state.premise is null` + - success: set `state.premise = sanitize_premise(X)` + - if premise already exists: `clarify` -For additive predicates: +- `change premise to X`: + - valid only if `state.premise is not null` + - success: replace premise with `sanitize_premise(X)` + - if no premise exists: `clarify` -- Split on commas and “and” -- Apply normalization to each item +### 8.2 Policy lifecycle -For exclusive predicates: +Let `k = normalize_item(ITEM)`. -- Multiple values → clarification +- `use ITEM`: + - if `policies[k] == "prohibit"`: `clarify` (contradiction) + - if `policies[k] == "use"`: no-op `update` (idempotent assertion) + - else set `policies[k] = "use"` -### 6. Normalization +- `don't use ITEM`: + - if `policies[k] == "use"`: `clarify` (contradiction) + - if `policies[k] == "prohibit"`: no-op `update` (idempotent assertion) + - else set `policies[k] = "prohibit"` -`normalize(X)` performs: +### 8.3 Explicit replacement -1. Unicode NFKC normalization -2. Lowercase -3. Collapse whitespace -4. Remove leading articles: a, an, the -5. Normalize apostrophes (`dont` → `don't`) +For `use X instead of Y`: -No stemming, synonym mapping, ontology, or semantic interpretation. +1. Let `kx = normalize_item(X)`, `ky = normalize_item(Y)`. +2. If `kx == ky`: no-op `update`. +3. Otherwise, `Y` must currently exist in policy state (`ky in policies`) or return `clarify`. +4. Replacement must not silently change polarity: + - if `policies[ky] != "use"`, return `clarify`. +5. Replacement must not create contradiction at target: + - if `policies.get(kx) == "prohibit"`, return `clarify`. +6. On success: + - remove `ky` from `policies` + - set `policies[kx] = "use"` -Policy payloads (add/remove) use `normalize(X)` exactly as defined above. +This operation is authoritative replacement, not recency resolution. -Fact values do not use `normalize(X)`. They may receive minimal input -sanitation only: +### 8.4 Administrative commands -1. Unicode normalization -2. Apostrophe normalization -3. Whitespace collapse +- `clear premise`: set `premise = null` (cleared premise state) +- `reset policies`: set `policies = {}` +- `clear state`: reset all authoritative state by setting `premise = null` and `policies = {}` -Fact sanitation does not include lowercasing or leading-article removal. +## 9. Clarification Rules (Exhaustive) -### 7. Ambiguity Handling +The compiler returns `Decision.kind = "clarify"` only in these cases: -If a message might mutate state but is unclear: +1. `set premise X` when a premise already exists. +2. `change premise to X` when no premise exists. +3. `use ITEM` when that item is currently `"prohibit"`. +4. `don't use ITEM` when that item is currently `"use"`. +5. `use X instead of Y` when `Y` does not exist in policies. +6. `use X instead of Y` when `Y` exists but is not `"use"`. +7. `use X instead of Y` when `X` is currently `"prohibit"`. +8. A pending clarification exists and input is not an exact confirmation token. - Decision.kind = "clarify" - Decision.prompt_to_user = clarification question +Contradictions never silently overwrite state. -This includes correction payloads that appear to invoke another -directive family. +### 9.1 Standardized clarify prompts -Examples: +When `Decision.kind = "clarify"`, prompt text is deterministic only for the cases listed below. -- "don use parallel octaves" -- "no use docker" +- Replacement-intent conflicts (Section 9 cases 5-7): + `Did you mean to replace "Y" with "X"?` +- Pending clarification unmatched input (Section 9 case 8): + reuse the existing pending prompt unchanged. -No state mutation occurs. +Clarify cases 1-4 must return `clarify` but do not require a standardized prompt string in this specification. +Their exact prompt text is implementation-defined unless standardized in a later spec revision. -Non-directive input is always treated as passthrough; the compiler does not evaluate usefulness. - -### 8. Pending Clarification +## 10. Pending Clarification Internal structure: -`python - pending = { - proposed_event - } -` - -Resolution: -| | | -|--------------|---------------| -| **Response** | Action | -| affirmative confirmation token | apply event | -| negative confirmation token | discard event | +```python +pending = { + "proposed_event": ..., + "prompt": ... +} +``` -Confirmation parsing takes precedence over all other directive parsing -while pending clarification exists. +While pending exists: -A pending clarification resolves only when normalized user input exactly -matches one of the confirmation tokens. +- Directive parsing is suspended. +- Only confirmation tokens are processed. +- Accepted affirmative tokens: `yes`, `yes please`, `yep`, `yeah`, `sure`, `ok`, `okay` +- Accepted negative tokens: `no`, `nope`, `no thanks` -Normalization for pending confirmation matching: +Confirmation token normalization: 1. Trim surrounding whitespace 2. Lowercase -3. Collapse internal whitespace to single spaces +3. Collapse internal whitespace 4. Remove trailing punctuation: `. , ! ?` -Accepted affirmative confirmation tokens: - -- `yes` -- `yes please` -- `yep` -- `yeah` -- `sure` -- `ok` -- `okay` - -Accepted negative confirmation tokens: - -- `no` -- `nope` -- `no thanks` - -If no pending exists → confirmation tokens are passthrough. -If normalized input does not match a confirmation token while pending -exists, the compiler remains in clarify, does not trigger other -directive parsing, and does not mutate state. - -### 9. State Update Semantics - -#### Facts (exclusive) - - focus.primary = "nord stage 3" - focus.primary = "nord stage 4" - -→` only stage 4 active` - -#### Policies (additive) - - don't use parallel octaves - do not use voice crossing - -Adding duplicate policy is a no-op. -Policies stored in sorted lexical order. - -Administrative state initialization/replacement is supported through: -- constructor input (`create_engine(state=...)` / `Engine(state=...)`) for initial load -- `engine.import_json(payload)` (JSON replacement) +Resolution: -Import-based replacement clears pending clarification state and must behave like -live state for subsequent `step()` calls. +- Affirmative token: apply pending event, clear pending, return `update` +- Negative token: discard pending event, clear pending, return `passthrough` +- Any other input: remain in `clarify`, no mutation, and keep the existing pending prompt -### 10. Context Serialization Contract +## 11. Context Serialization Contract -The compiler outputs structured state only. The host formats prompts. +The compiler outputs structured state only; the host formats prompts. Example: -`json - { - "facts": {"focus.primary": "nord stage 4"}, - "policies": {"prohibit": ["parallel octaves"]} - } -` +```json +{ + "premise": "Use concise, formal language.", + "policies": { + "docker": "prohibit", + "pytest": "use" + }, + "version": 2 +} +``` JSON persistence boundary: -- `engine.export_json()` serializes authoritative state for transport/storage. -- `engine.import_json(payload)` validates/canonicalizes payload and fully replaces active state. -### 11. Reset Commands +- `engine.export_json()` serializes authoritative state. +- `engine.import_json(payload)` validates/canonicalizes payload and replaces active state. -Explicit only: +## 12. Invariants -- "reset policies" -- "clear state" +1. State changes only from valid directive transitions or pending-confirmation acceptance. +2. Same input sequence yields identical state and decisions. +3. LLM output never mutates state. +4. No mutation occurs when returning `clarify`. +5. Premise updates are explicit and lifecycle-gated (`set` vs `change`). +6. Policy state is per-item authoritative (`"use" | "prohibit"`), never ordered/additive by history. +7. Contradictions always clarify; they never overwrite. +8. Premise can be explicitly cleared via `clear premise` (`premise = null`). -Produces: - - if command == "reset policies": - state.policies.prohibit = [] - elif command == "clear state": - state = initial_state - Decision.kind = "update" - -### 12. Non-Goals +## 13. Non-Goals Not implemented: -- reference resolution +- `has` / `have` parsing +- `is`-parsing +- entity modeling +- ordered policies +- dict+history hybrids +- natural-language understanding inside the compiler - cross-session memory -- logical contradiction detection -- ontology reasoning (Docker Ë container) -- scoped overrides -- output validation +- ontology reasoning - agent planning - -### 13. Invariants - -1. State never changes without high-confidence directive or confirmation -2. Same input sequence → identical state -3. LLM output never affects state -4. No mutation occurs during `clarify` -5. Facts are exclusive; policies are additive -6. Pending clarification blocks mutation - -### 14. Property Tests (Required) - -Examples: - -**Determinism** Same sequence → same state -**Safety** Near-miss input never mutates state -**Replacement** Later fact overwrites earlier fact -**Persistence** Policies persist across turns -**Idempotency** Duplicate policy adds do nothing +- output validation ## End -This milestone establishes the authoritative truth layer.All later -milestones build on this deterministic contract. +Version 0.5 removes M1 recency and additive-policy semantics in favor of +explicit premise lifecycle and authoritative per-item policy state.