Skip to content

docs: document automatic skill-load marks [3/3] #393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
willkill07 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: document automatic skill-load marks [3/3] #393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions docs/about-nemo-relay/concepts/events.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,53 @@ streams are blocked, or how exporter-specific semantic projection works. Those
decisions stay in the session, codec, plugin, guardrail, or exporter layer that
owns them.

### Automatic Skill-Load Marks

NeMo Relay automatically records an on-demand skill load when an instrumented
tool loads a complete `SKILL.md` file. The observed mark contract is:

```json
{
"kind": "mark",
"name": "skill.load",
"data": {"skill_name": "review"},
"metadata": {
"skill_load_source": "structured_read",
"tool_name": "read_file"
}
}
```
Comment on lines +117 to +127

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Make the “exact” event example include its envelope fields.

The implementation also sets parent_uuid and timestamp on each mark, but the example omits them while presenting itself as the complete observed shape. Either include those fields with their serialized types or label this example as the payload-only shape.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/about-nemo-relay/concepts/events.mdx` around lines 117 - 127, Update the
exact event example in the events documentation to include the envelope fields
parent_uuid and timestamp with their serialized types, matching the
implementation’s mark event shape; alternatively, explicitly label the example
as payload-only rather than complete.


`skill_load_source` is `skill_tool`, `structured_read`, or `shell_read`. Relay
recognizes first-class skill tools, whole-file structured readers, and
standalone complete shell readers such as `cat`, `bat`, and PowerShell
`Get-Content`. The path must end in `SKILL.md`; the skill name is its immediate
parent directory. Structured offsets or limits and shell pipelines,
redirections, compound commands, and range-limited readers do not count. Relay
does not put the full path or shell command in the mark.

The mark is emitted immediately after tool start and before execution, with the
tool span as its parent. It records an eager load attempt, so it remains present
if the read later fails. A blocked call that never starts does not emit the
mark. Each skill is emitted once within one tool call; a later call that loads
the same skill emits another mark.

Claude Code `Skill`, Codex complete-reader shell calls, Hermes `skill_view`, and
tool calls made through the Rust, Python, Node.js, Go/FFI, LangChain,
LangGraph, and Deep Agents paths use this observed contract. The existing Deep
Agents `DeepAgents Skills Configured` mark remains a separate configuration
summary; it does not mean that a skill body was loaded.

Claude Code exposes direct slash-command expansion without saying whether the
command came from a skill or a legacy command. Relay emits those nonempty
expansions as `skill.load.inferred`, with
`data.skill_name`, `metadata.skill_load_source = "prompt_expansion"`, and host
metadata. Consumers should keep inferred and observed counts separate.

ATOF preserves the mark exactly. ATIF omits marks because it models trajectory
steps rather than independent point-in-time events. OpenTelemetry and
OpenInference use their configured mark projection behavior for parented marks.

## How Events Are Produced

Scope APIs emit `scope` start and end events and can also emit named `mark`
Expand Down
2 changes: 1 addition & 1 deletion docs/nemo-relay-cli/basic-usage.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ Generated hook bundles subscribe to the events needed for that mapping:

| Agent | LLM lifecycle and correlation hooks | Scope, tool, and mark hooks |
| --- | --- | --- |
| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` |
| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact`, `UserPromptExpansion` |
| Codex | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` |
| Hermes | `pre_api_request`, `post_api_request`, `api_request_error`, `pre_llm_call`, `post_llm_call` | `on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`, `subagent_start`, `subagent_stop`, `pre_tool_call`, `post_tool_call` |

Expand Down
5 changes: 4 additions & 1 deletion docs/nemo-relay-cli/claude-code.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,10 @@ process.
Generated Claude Code hooks include `SessionStart`, `SessionEnd`,
`SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`,
`PostToolUseFailure`, `Notification`, and `PreCompact` for scope, tool, and
mark events. `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and
mark events. `UserPromptExpansion` records nonempty slash-command expansions
as `skill.load.inferred`; Claude Code does not identify whether that expansion
came from a skill or a legacy custom command. The normal `Skill` pre-tool hook
emits an observed `skill.load` mark. `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and
`Stop` are retained as private LLM correlation hints and are not emitted as
standalone NeMo Relay events.

Expand Down
4 changes: 3 additions & 1 deletion docs/nemo-relay-cli/codex.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,9 @@ traffic to pass through the gateway.

Generated Codex hooks include `SessionStart`, `SessionEnd`, `SubagentStart`,
`SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`,
`Notification`, and `PreCompact` for scope, tool, and mark events.
`Notification`, and `PreCompact` for scope, tool, and mark events. Complete
standalone `cat`, `bat`, or PowerShell `Get-Content` reads of `SKILL.md` emit
observed `skill.load` marks.
`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are
retained as private LLM correlation hints and are not emitted as standalone
NeMo Relay events.
Expand Down
3 changes: 3 additions & 0 deletions docs/nemo-relay-cli/hermes.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,9 @@ The generated Hermes hooks cover `on_session_start`, `on_session_end`,
`pre_api_request`, `post_api_request`, `api_request_error`, `pre_tool_call`,
`post_tool_call`, `subagent_start`, and `subagent_stop`.

Hermes `skill_view` pre-tool calls emit observed `skill.load` marks. The mark
contains only the skill name and tool identity; it does not retain skill paths.
Comment on lines +131 to +132

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Distinguish the payload from the complete mark metadata.

The canonical mark contains only skill_name in data, but its metadata also includes skill_load_source and tool_name. Rewrite this as “the payload contains only the skill name; metadata records the load source and tool name,” while retaining the statement that paths are omitted.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/nemo-relay-cli/hermes.mdx` around lines 131 - 132, The documentation
should distinguish the mark payload from its metadata: rewrite the Hermes
`skill_view` description so `data` is said to contain only `skill_name`, while
metadata records `skill_load_source` and `tool_name`; retain that skill paths
are omitted.


The API hooks are the main Hermes LLM lifecycle path for NeMo Relay. The legacy
LLM hooks remain installed because they can still provide useful private hints,
but they are not treated as equal peers to the API hooks in the observability
Expand Down
3 changes: 3 additions & 0 deletions docs/supported-integrations/about.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,8 @@ instructions.
Agents extra and capturing Deep Agents-specific marks, skills, subagents, and
human-in-the-loop lifecycle events.

Across LangChain, LangGraph, and Deep Agents, instrumented tool lifecycles also
inherit Relay's automatic `skill.load` detection for complete `SKILL.md` reads.

If you are building a new public-API framework integration,
use [Integrate into Frameworks](/integrate-into-frameworks/about) instead.
2 changes: 2 additions & 0 deletions docs/supported-integrations/deepagents.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@ It captures:
- LangGraph run scopes through callbacks.
- Human-in-the-loop interrupt and resume marks.
- Configured skills and subagent summaries at agent-run start.
- Automatic `skill.load` marks when a Deep Agents tool reads a complete
`SKILL.md`; this is distinct from the configured-skills summary.
- In-process dictionary-style subagents with the same NeMo Relay middleware, so
their model and tool calls are captured when Deep Agents invokes them.

Expand Down
4 changes: 4 additions & 0 deletions skills/nemo-relay-setup-observability/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ activity but has not yet decided which output they need.
- Event fields include semantic input/output through the ATOF `data` field,
typed profile data such as `model_name` and `tool_call_id`, and codec-provided
annotated LLM request/response data for in-process subscribers and exporters.
- Complete on-demand `SKILL.md` loads automatically emit `skill.load` marks
under the tool span. The payload contains only `skill_name`; metadata records
the load source and tool name. Partial reads do not count, and ambiguous
slash-command expansions use the separate `skill.load.inferred` name.

## Shared Lifecycle

Expand Down
Loading