Skip to content

docs: document LLM event freshness behavior #406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
willkill07 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

docs: document LLM event freshness behavior #406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4f5148c
docs: document LLM event freshness behavior
willkill07 Jul 11, 2026
c54cc50
Merge branch 'main' into docs/relay-445-event-freshness
willkill07 Jul 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion docs/about-nemo-relay/concepts/events.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,20 @@ ATOF uses one `data` field. For scope events, `data` is the semantic input on
`scope_category == "start"` and the semantic output on
`scope_category == "end"`.

LLM start annotations retain complete sanitized request history while their
owning agent session is fresh: initially and after a compaction mark. Once the
agent begins a turn, later annotations retain system instructions, the latest
user message, and every following assistant or tool message. Nested agents
track freshness independently. The implicit root is an agent freshness owner.
Non-agent scopes, including function and custom scopes used to model requests,
inherit their nearest agent's freshness instead of starting a new budget. When
a request codec is active, it projects the event's provider-shaped input to the
same history. If that event-only encode fails, Relay retains the complete event
input and annotation and continues the provider call. The request used for
provider execution remains unchanged. Concurrent starts have no guaranteed
emission order, but exactly one consumes a fresh state. Events are independent
observations rather than a replayable history stream.

Comment on lines +83 to +96

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Dense paragraph mixes six distinct concepts.

This paragraph packs freshness definition, turn-based retention, nested-agent independence, non-agent scope inheritance, codec projection/fallback, and concurrent-start semantics into one block. Splitting into a short bulleted list (one idea per item) would make this easier to scan.

📝 Suggested restructure 
-LLM start annotations retain complete sanitized request history while their
-owning agent session is fresh: initially and after a compaction mark. Once the
-agent begins a turn, later annotations retain system instructions, the latest
-user message, and every following assistant or tool message. Nested agents
-track freshness independently. The implicit root is an agent freshness owner.
-Non-agent scopes, including function and custom scopes used to model requests,
-inherit their nearest agent's freshness instead of starting a new budget. When
-a request codec is active, it projects the event's provider-shaped input to the
-same history. If that event-only encode fails, Relay retains the complete event
-input and annotation and continues the provider call. The request used for
-provider execution remains unchanged. Concurrent starts have no guaranteed
-emission order, but exactly one consumes a fresh state. Events are independent
-observations rather than a replayable history stream.
+LLM start annotations retain complete sanitized request history while their
+owning agent session is fresh: initially and after a compaction mark. Once the
+agent begins a turn, later annotations retain system instructions, the latest
+user message, and every following assistant or tool message.
+
+- Nested agents track freshness independently, and the implicit root is an
+  agent freshness owner.
+- Non-agent scopes, including function and custom scopes used to model
+  requests, inherit their nearest agent's freshness instead of starting a new
+  budget.
+- When a request codec is active, it projects the event's provider-shaped
+  input to the same history. If that event-only encode fails, Relay retains
+  the complete event input and annotation and continues the provider call.
+  The request used for provider execution remains unchanged.
+- Concurrent starts have no guaranteed emission order, but exactly one
+  consumes a fresh state. Events are independent observations rather than a
+  replayable history stream.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
LLM start annotations retain complete sanitized request history while their
owning agent session is fresh: initially and after a compaction mark. Once the
agent begins a turn, later annotations retain system instructions, the latest
user message, and every following assistant or tool message. Nested agents
track freshness independently. The implicit root is an agent freshness owner.
Non-agent scopes, including function and custom scopes used to model requests,
inherit their nearest agent's freshness instead of starting a new budget. When
a request codec is active, it projects the event's provider-shaped input to the
same history. If that event-only encode fails, Relay retains the complete event
input and annotation and continues the provider call. The request used for
provider execution remains unchanged. Concurrent starts have no guaranteed
emission order, but exactly one consumes a fresh state. Events are independent
observations rather than a replayable history stream.
LLM start annotations retain complete sanitized request history while their
owning agent session is fresh: initially and after a compaction mark. Once the
agent begins a turn, later annotations retain system instructions, the latest
user message, and every following assistant or tool message.
- Nested agents track freshness independently, and the implicit root is an
agent freshness owner.
- Non-agent scopes, including function and custom scopes used to model
requests, inherit their nearest agent's freshness instead of starting a new
budget.
- When a request codec is active, it projects the event's provider-shaped
input to the same history. If that event-only encode fails, Relay retains
the complete event input and annotation and continues the provider call.
The request used for provider execution remains unchanged.
- Concurrent starts have no guaranteed emission order, but exactly one
consumes a fresh state. Events are independent observations rather than a
replayable history stream.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/about-nemo-relay/concepts/events.mdx` around lines 83 - 96, Restructure
the dense paragraph about LLM start annotations into a short bulleted list, with
one item each for freshness and retention, nested-agent independence, non-agent
scope inheritance, codec projection and fallback, concurrent-start behavior, and
event independence. Preserve all existing semantics and details while making
each concept independently scannable.

### Category Profiles

Category-specific fields live under `category_profile`. NeMo Relay uses
Expand All @@ -94,7 +108,7 @@ are preserved so newer producers can interoperate with older consumers.
LLM codecs can enrich LLM events with annotated request and response data. These
annotations are part of the canonical event JSON under `category_profile` when
they are present, so ATOF JSONL export and in-process subscriber JSON expose the
same payload shape.
same payload shape. Request annotations use the owning agent's freshness state.

### Event Contract Boundary

Expand Down
6 changes: 5 additions & 1 deletion docs/configure-plugins/observability/atof.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,11 @@ JSON object per line. For event field semantics, refer to

ATOF is the raw-event export path. It preserves the canonical event shape; it
does not add semantic extraction, replay policy, scope-owner resolution, or
exporter projection rules.
exporter projection rules. LLM request annotations retain complete history
while their agent is fresh. When stale, they retain system instructions, the
latest user message, and every following assistant or tool message. With a
request codec, the emitted event input uses the same projection. The request
used for provider execution remains unchanged.

Register the plugin before instrumented work starts and clear it during
shutdown so file handles flush.
Expand Down
9 changes: 8 additions & 1 deletion docs/instrument-applications/instrument-llm-call.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,14 @@ Native subscriber delivery is asynchronous. Flush subscribers before validating
printed output. In Node.js, also wait one event-loop tick after
`flushSubscribers()` so JavaScript callbacks can run.
- LLM start input contains the request after request intercepts and
sanitize-request guardrails.
sanitize-request guardrails. Annotations retain complete history while their
agent session is fresh (initially and after compaction); later events contain
the current user turn. With a request codec, the emitted event input uses the
same projection. This event-only projection does not change the request used
for provider execution.
Comment on lines +241 to +245

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Stale-session retention description is inconsistent with events.mdx and atof.mdx.

This says stale annotations "contain the current user turn," but events.mdx (lines 85-86) and atof.mdx (lines 105-107) both specify that stale annotations retain system instructions, the latest user message, and subsequent assistant/tool messages. As written here, a reader could conclude the system prompt is dropped once a turn begins.

📝 Suggested wording fix 
-  sanitize-request guardrails. Annotations retain complete history while their
-  agent session is fresh (initially and after compaction); later events contain
-  the current user turn. With a request codec, the emitted event input uses the
+  sanitize-request guardrails. Annotations retain complete history while their
+  agent session is fresh (initially and after compaction); later events retain
+  system instructions, the latest user message, and subsequent assistant or
+  tool messages. With a request codec, the emitted event input uses the
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
sanitize-request guardrails. Annotations retain complete history while their
agent session is fresh (initially and after compaction); later events contain
the current user turn. With a request codec, the emitted event input uses the
same projection. This event-only projection does not change the request used
for provider execution.
sanitize-request guardrails. Annotations retain complete history while their
agent session is fresh (initially and after compaction); later events retain
system instructions, the latest user message, and subsequent assistant or
tool messages. With a request codec, the emitted event input uses the
same projection. This event-only projection does not change the request used
for provider execution.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/instrument-applications/instrument-llm-call.mdx` around lines 241 - 245,
Update the stale-session annotation description near “sanitize-request
guardrails” to match events.mdx and atof.mdx: state that stale annotations
retain system instructions, the latest user message, and subsequent
assistant/tool messages, rather than only the current user turn. Preserve the
distinction that fresh sessions retain complete history and that the request
codec applies the same projection to emitted event input.

- Freshness belongs to agent scopes, including the implicit root. Non-agent
scopes, including function and custom scopes used to model requests, inherit
their nearest agent's state and do not start a new freshness budget.
- LLM end output contains the provider response after response guardrails.
- The LLM event includes the normalized `model_name` when you provide one.

Expand Down
4 changes: 2 additions & 2 deletions docs/nemo-relay-cli/basic-usage.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,8 +306,8 @@ Generated hook bundles subscribe to the events needed for that mapping:

| Agent | LLM lifecycle and correlation hooks | Scope, tool, and mark hooks |
| --- | --- | --- |
| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` |
| Codex | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` |
| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact`, `PostCompact` |
| Codex | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact`, `PostCompact` |
| Hermes | `pre_api_request`, `post_api_request`, `api_request_error`, `pre_llm_call`, `post_llm_call` | `on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`, `subagent_start`, `subagent_stop`, `pre_tool_call`, `post_tool_call` |

Hermes `pre_api_request`, `post_api_request`, and `api_request_error` hooks
Expand Down
9 changes: 5 additions & 4 deletions docs/nemo-relay-cli/claude-code.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,10 +122,11 @@ process.

Generated Claude Code hooks include `SessionStart`, `SessionEnd`,
`SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`,
`PostToolUseFailure`, `Notification`, and `PreCompact` for scope, tool, and
mark events. `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and
`Stop` are retained as private LLM correlation hints and are not emitted as
standalone NeMo Relay events.
`PostToolUseFailure`, `Notification`, `PreCompact`, and `PostCompact` for scope,
tool, and mark events. Both compaction hooks emit canonical `compaction` marks.
`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are
retained as private LLM correlation hints and are not emitted as standalone
NeMo Relay events.

Tool hooks preserve canonical fields such as `tool_use_id`, `tool_name`,
`tool_input`, `error`, `duration_ms`, and `is_interrupt`. Subagent hooks use
Expand Down
3 changes: 2 additions & 1 deletion docs/nemo-relay-cli/codex.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,8 @@ traffic to pass through the gateway.

Generated Codex hooks include `SessionStart`, `SessionEnd`, `SubagentStart`,
`SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`,
`Notification`, and `PreCompact` for scope, tool, and mark events.
`Notification`, `PreCompact`, and `PostCompact` for scope, tool, and mark
events. Both compaction hooks emit canonical `compaction` marks.
`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are
retained as private LLM correlation hints and are not emitted as standalone
NeMo Relay events.
Expand Down
6 changes: 5 additions & 1 deletion skills/nemo-relay-export-atif-trajectories/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,11 @@ live OTLP spans.

- ATIF export translates NeMo Relay events into ATIF v1.7 trajectory data.
- LLM start events become `user` steps; message content is extracted from the
`LLMRequest.content` payload when possible.
request annotation when possible. Annotations retain complete history while
their agent session is fresh (initially and after compaction); otherwise, the
projection retains system instructions, the latest user message, and following
assistant or tool messages. With a request codec, the event input uses the same
projection; provider execution remains unchanged.
Comment on lines +26 to +30

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Clarify annotation history versus exported ATIF content.

The ATIF adapter in crates/core/src/observability/atif.rs:1040-1105 currently projects only the latest User message into the ATIF message; it does not emit the complete annotated history. Clarify that full history is retained by the annotation, while the ATIF user-step projection is narrower.

Suggested wording 
-  request annotation when possible. Annotations retain complete history while
+  request annotation when possible. The annotation retains complete history while
   their agent session is fresh (initially and after compaction); otherwise, the
-  projection retains system instructions, the latest user message, and following
-  assistant or tool messages. With a request codec, the event input uses the same
-  projection; provider execution remains unchanged.
+  annotation projection retains system instructions, the latest user message, and
+  following assistant or tool messages. The ATIF user-step projection extracts
+  the latest user message. With a request codec, the event input uses the same
+  annotation projection; provider execution remains unchanged.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-export-atif-trajectories/SKILL.md` around lines 26 - 30,
Clarify the documentation near the annotation and projection behavior to
distinguish retained annotation history from exported ATIF content: annotations
preserve complete history when available, while the ATIF adapter’s user-step
projection, implemented in the relevant ATIF conversion logic, includes only the
latest User message (or the documented fallback projection). State that request
codecs affect event input projection without changing provider execution.

- LLM end events become `agent` steps with response content, model metadata,
token metrics, reasoning fields, and promoted `tool_calls` when the response
uses a supported tool-call shape.
Expand Down
5 changes: 5 additions & 0 deletions skills/nemo-relay-export-openinference/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,11 @@ for example Arize Phoenix or another OpenInference-aware OTLP backend.
`grpc` only when a Tokio runtime is active.
- Scope, tool, and LLM start inputs become `input.value`.
- Scope, tool, and LLM end outputs become `output.value`.
- Annotated LLM input retains complete sanitized history while the agent session
is fresh (initially and after compaction). When stale, it retains system
instructions, the latest user message, and every following assistant or tool
message. With a request codec, the event's provider-shaped input uses the same
projection; provider execution remains unchanged.
- LLM usage metadata maps token counters when provider responses include usage.
- Use explicit config fields for endpoint, headers, resource attributes, and
service identity in application code.
Expand Down
5 changes: 5 additions & 0 deletions skills/nemo-relay-setup-observability/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ activity but has not yet decided which output they need.
OpenTelemetry, or OpenInference output.
- Event payloads reflect sanitized post-guardrail input and output when calls use
managed helpers or manual lifecycle params provide those fields.
- Annotated LLM start histories retain complete sanitized context while their
agent session is fresh (initially and after compaction). When stale,
annotations retain system instructions, the latest user message, and every
following assistant or tool message. Codec-backed event inputs use the same
projection; provider execution remains unchanged.
- Event fields include semantic input/output through the ATOF `data` field,
typed profile data such as `model_name` and `tool_call_id`, and codec-provided
annotated LLM request/response data for in-process subscribers and exporters.
Expand Down
Loading