From 4f5148ca92c0f43cae492b955818ee1f35c3558b Mon Sep 17 00:00:00 2001 From: Will Killian Date: Fri, 10 Jul 2026 20:04:21 -0400 Subject: [PATCH] docs: document LLM event freshness behavior Signed-off-by: Will Killian --- docs/about-nemo-relay/concepts/events.mdx | 16 +++++++++++++++- docs/configure-plugins/observability/atof.mdx | 6 +++++- .../instrument-llm-call.mdx | 9 ++++++++- docs/nemo-relay-cli/basic-usage.mdx | 4 ++-- docs/nemo-relay-cli/claude-code.mdx | 9 +++++---- docs/nemo-relay-cli/codex.mdx | 3 ++- .../nemo-relay-export-atif-trajectories/SKILL.md | 6 +++++- skills/nemo-relay-export-openinference/SKILL.md | 5 +++++ skills/nemo-relay-setup-observability/SKILL.md | 5 +++++ 9 files changed, 52 insertions(+), 11 deletions(-) diff --git a/docs/about-nemo-relay/concepts/events.mdx b/docs/about-nemo-relay/concepts/events.mdx index dddf27cd0..a67fd428f 100644 --- a/docs/about-nemo-relay/concepts/events.mdx +++ b/docs/about-nemo-relay/concepts/events.mdx @@ -80,6 +80,20 @@ ATOF uses one `data` field. For scope events, `data` is the semantic input on `scope_category == "start"` and the semantic output on `scope_category == "end"`. +LLM start annotations retain complete sanitized request history while their +owning agent session is fresh: initially and after a compaction mark. Once the +agent begins a turn, later annotations retain system instructions, the latest +user message, and every following assistant or tool message. Nested agents +track freshness independently. The implicit root is an agent freshness owner. +Non-agent scopes, including function and custom scopes used to model requests, +inherit their nearest agent's freshness instead of starting a new budget. When +a request codec is active, it projects the event's provider-shaped input to the +same history. If that event-only encode fails, Relay retains the complete event +input and annotation and continues the provider call. The request used for +provider execution remains unchanged. Concurrent starts have no guaranteed +emission order, but exactly one consumes a fresh state. Events are independent +observations rather than a replayable history stream. + ### Category Profiles Category-specific fields live under `category_profile`. NeMo Relay uses @@ -94,7 +108,7 @@ are preserved so newer producers can interoperate with older consumers. LLM codecs can enrich LLM events with annotated request and response data. These annotations are part of the canonical event JSON under `category_profile` when they are present, so ATOF JSONL export and in-process subscriber JSON expose the -same payload shape. +same payload shape. Request annotations use the owning agent's freshness state. ### Event Contract Boundary diff --git a/docs/configure-plugins/observability/atof.mdx b/docs/configure-plugins/observability/atof.mdx index d0103354e..51f3075c5 100644 --- a/docs/configure-plugins/observability/atof.mdx +++ b/docs/configure-plugins/observability/atof.mdx @@ -101,7 +101,11 @@ JSON object per line. For event field semantics, refer to ATOF is the raw-event export path. It preserves the canonical event shape; it does not add semantic extraction, replay policy, scope-owner resolution, or -exporter projection rules. +exporter projection rules. LLM request annotations retain complete history +while their agent is fresh. When stale, they retain system instructions, the +latest user message, and every following assistant or tool message. With a +request codec, the emitted event input uses the same projection. The request +used for provider execution remains unchanged. Register the plugin before instrumented work starts and clear it during shutdown so file handles flush. diff --git a/docs/instrument-applications/instrument-llm-call.mdx b/docs/instrument-applications/instrument-llm-call.mdx index 0d75aadcf..91df8da13 100644 --- a/docs/instrument-applications/instrument-llm-call.mdx +++ b/docs/instrument-applications/instrument-llm-call.mdx @@ -238,7 +238,14 @@ Native subscriber delivery is asynchronous. Flush subscribers before validating printed output. In Node.js, also wait one event-loop tick after `flushSubscribers()` so JavaScript callbacks can run. - LLM start input contains the request after request intercepts and - sanitize-request guardrails. + sanitize-request guardrails. Annotations retain complete history while their + agent session is fresh (initially and after compaction); later events contain + the current user turn. With a request codec, the emitted event input uses the + same projection. This event-only projection does not change the request used + for provider execution. +- Freshness belongs to agent scopes, including the implicit root. Non-agent + scopes, including function and custom scopes used to model requests, inherit + their nearest agent's state and do not start a new freshness budget. - LLM end output contains the provider response after response guardrails. - The LLM event includes the normalized `model_name` when you provide one. diff --git a/docs/nemo-relay-cli/basic-usage.mdx b/docs/nemo-relay-cli/basic-usage.mdx index 674ebed8b..b8acace1a 100644 --- a/docs/nemo-relay-cli/basic-usage.mdx +++ b/docs/nemo-relay-cli/basic-usage.mdx @@ -306,8 +306,8 @@ Generated hook bundles subscribe to the events needed for that mapping: | Agent | LLM lifecycle and correlation hooks | Scope, tool, and mark hooks | | --- | --- | --- | -| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` | -| Codex | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` | +| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact`, `PostCompact` | +| Codex | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact`, `PostCompact` | | Hermes | `pre_api_request`, `post_api_request`, `api_request_error`, `pre_llm_call`, `post_llm_call` | `on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`, `subagent_start`, `subagent_stop`, `pre_tool_call`, `post_tool_call` | Hermes `pre_api_request`, `post_api_request`, and `api_request_error` hooks diff --git a/docs/nemo-relay-cli/claude-code.mdx b/docs/nemo-relay-cli/claude-code.mdx index abc80726a..243e42f82 100644 --- a/docs/nemo-relay-cli/claude-code.mdx +++ b/docs/nemo-relay-cli/claude-code.mdx @@ -122,10 +122,11 @@ process. Generated Claude Code hooks include `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, -`PostToolUseFailure`, `Notification`, and `PreCompact` for scope, tool, and -mark events. `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and -`Stop` are retained as private LLM correlation hints and are not emitted as -standalone NeMo Relay events. +`PostToolUseFailure`, `Notification`, `PreCompact`, and `PostCompact` for scope, +tool, and mark events. Both compaction hooks emit canonical `compaction` marks. +`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are +retained as private LLM correlation hints and are not emitted as standalone +NeMo Relay events. Tool hooks preserve canonical fields such as `tool_use_id`, `tool_name`, `tool_input`, `error`, `duration_ms`, and `is_interrupt`. Subagent hooks use diff --git a/docs/nemo-relay-cli/codex.mdx b/docs/nemo-relay-cli/codex.mdx index 2446e1ff4..e83f651d9 100644 --- a/docs/nemo-relay-cli/codex.mdx +++ b/docs/nemo-relay-cli/codex.mdx @@ -181,7 +181,8 @@ traffic to pass through the gateway. Generated Codex hooks include `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, -`Notification`, and `PreCompact` for scope, tool, and mark events. +`Notification`, `PreCompact`, and `PostCompact` for scope, tool, and mark +events. Both compaction hooks emit canonical `compaction` marks. `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are retained as private LLM correlation hints and are not emitted as standalone NeMo Relay events. diff --git a/skills/nemo-relay-export-atif-trajectories/SKILL.md b/skills/nemo-relay-export-atif-trajectories/SKILL.md index 73cceb6f8..a8dd6995d 100644 --- a/skills/nemo-relay-export-atif-trajectories/SKILL.md +++ b/skills/nemo-relay-export-atif-trajectories/SKILL.md @@ -23,7 +23,11 @@ live OTLP spans. - ATIF export translates NeMo Relay events into ATIF v1.7 trajectory data. - LLM start events become `user` steps; message content is extracted from the - `LLMRequest.content` payload when possible. + request annotation when possible. Annotations retain complete history while + their agent session is fresh (initially and after compaction); otherwise, the + projection retains system instructions, the latest user message, and following + assistant or tool messages. With a request codec, the event input uses the same + projection; provider execution remains unchanged. - LLM end events become `agent` steps with response content, model metadata, token metrics, reasoning fields, and promoted `tool_calls` when the response uses a supported tool-call shape. diff --git a/skills/nemo-relay-export-openinference/SKILL.md b/skills/nemo-relay-export-openinference/SKILL.md index 3a69d4879..5e634c621 100644 --- a/skills/nemo-relay-export-openinference/SKILL.md +++ b/skills/nemo-relay-export-openinference/SKILL.md @@ -29,6 +29,11 @@ for example Arize Phoenix or another OpenInference-aware OTLP backend. `grpc` only when a Tokio runtime is active. - Scope, tool, and LLM start inputs become `input.value`. - Scope, tool, and LLM end outputs become `output.value`. +- Annotated LLM input retains complete sanitized history while the agent session + is fresh (initially and after compaction). When stale, it retains system + instructions, the latest user message, and every following assistant or tool + message. With a request codec, the event's provider-shaped input uses the same + projection; provider execution remains unchanged. - LLM usage metadata maps token counters when provider responses include usage. - Use explicit config fields for endpoint, headers, resource attributes, and service identity in application code. diff --git a/skills/nemo-relay-setup-observability/SKILL.md b/skills/nemo-relay-setup-observability/SKILL.md index 8812540f0..be7b16a11 100644 --- a/skills/nemo-relay-setup-observability/SKILL.md +++ b/skills/nemo-relay-setup-observability/SKILL.md @@ -38,6 +38,11 @@ activity but has not yet decided which output they need. OpenTelemetry, or OpenInference output. - Event payloads reflect sanitized post-guardrail input and output when calls use managed helpers or manual lifecycle params provide those fields. +- Annotated LLM start histories retain complete sanitized context while their + agent session is fresh (initially and after compaction). When stale, + annotations retain system instructions, the latest user message, and every + following assistant or tool message. Codec-backed event inputs use the same + projection; provider execution remains unchanged. - Event fields include semantic input/output through the ATOF `data` field, typed profile data such as `model_name` and `tool_call_id`, and codec-provided annotated LLM request/response data for in-process subscribers and exporters.