feat: add extensible LLM optimization accounting#389
Conversation
WalkthroughAdds optimization contribution and summary contracts, records and finalizes call-scoped optimization evidence, wires managed and streaming execution to emit it, projects it into observability, and extends FFI, Node, Python, Go, plugin, and worker bindings and tests. ChangesLLM Optimization Evidence Pipeline
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant llm_call_execute
participant LlmHandle
participant LlmOptimizationRecorder
participant Stream
Caller->>llm_call_execute: request
llm_call_execute->>LlmHandle: store recorder and scope stack
llm_call_execute->>LlmOptimizationRecorder: emit optimization marks
llm_call_execute->>Stream: scope_llm_optimization_recorder(...)
Stream-->>llm_call_execute: annotated response
llm_call_execute->>LlmOptimizationRecorder: finalize_optimization_summary
llm_call_execute-->>Caller: AnnotatedLlmResponse
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
github-actions
Bot
added
Feature
bbednarski9
force-pushed
the
bbednarski/llm-optimization-accounting
branch
from
dbca7ef to
9b7b3fd
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
crates/python/src/py_types/codecs.rs (1)
511-547: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winExpose
optimization_summaryonPyAnnotatedLLMResponse. Runtime-produced responses can carry this field, but the Python wrapper never exposes it, so Python callers can’t read optimization data from decoded responses. Add a getter for parity with the core response type.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/python/src/py_types/codecs.rs` around lines 511 - 547, Expose the missing optimization data in PyAnnotatedLLMResponse so Python can read runtime-produced optimization details. Update the PyAnnotatedLLMResponse wrapper around AnnotatedLLMResponse to add a getter for optimization_summary, matching the core response type’s exposed fields and keeping parity with the existing message/tool_calls/usage accessors.Source: Path instructions
crates/ffi/src/api/mod.rs (1)
261-323: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winAdd
optimization_contributionsto the FFI outcome constructor.crates/ffi/src/api/mod.rs:261-323
nemo_relay_llm_request_intercept_outcome_json_newalways serializesoptimization_contributions: Vec::new(), while the Python and Node constructors accept caller-supplied contributions. Add an optional JSON input here too, or this FFI surface will silently drop that part of the intercept outcome contract.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/ffi/src/api/mod.rs` around lines 261 - 323, `nemo_relay_llm_request_intercept_outcome_json_new` is dropping caller-provided optimization contributions by always setting `optimization_contributions` to an empty vector. Update this constructor to accept an additional optional JSON input for contributions, parse it alongside `annotated_json` and `pending_marks_json`, and pass the parsed value into `LlmRequestInterceptOutcome` so the FFI API matches the Python and Node constructors.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/core/src/api/optimization.rs`:
- Around line 47-90: `LlmOptimizationRecorder::record` only bounds `payload`,
but `extra`, `model_transition`, and `token_impact` can still carry unbounded
data and bypass the intended limits. Update `record()` in `optimization.rs` to
measure and enforce a size cap over the full `LlmOptimizationContribution`
(including `extra` and nested strings) before accepting it, and reject/flag
oversized contributions using the existing `contribution_limit_exceeded` path.
In `@crates/core/src/observability/openinference.rs`:
- Around line 989-999: The pricing provenance lookup in the OpenInference
exporter is selecting a single provenance object too early, so a `baseline_cost`
without metadata hides valid `pricing_source` and `pricing_as_of` values from
`actual_cost`. Update the provenance handling in `openinference.rs` so each
field falls back independently from `baseline_cost` to `actual_cost`, and mirror
the same fix in the shared helper in `otel.rs` to keep the exporter behavior in
sync.
- Around line 957-981: The exported cost totals in openinference are missing
per-value currency context, so baseline_cost and actual_cost are ambiguous when
their currencies differ or savings is unavailable. Update the cost attribute
emission near the costs loop to attach the currency for each cost object
alongside the numeric total, using the existing summary fields and any
cost-specific currency metadata available from baseline_cost and actual_cost.
Keep the OpenInference cost export aligned with the corresponding ATIF and
OpenTelemetry implementations so the emitted event model stays consistent.
In `@crates/core/tests/integration/middleware_tests.rs`:
- Around line 3431-3542: The current test only covers a single sequential
request, so it doesn’t verify that optimization recording stays isolated per
call when requests run concurrently. Add a concurrent two-call test around the
existing test_managed_llm_materializes_optimization_mark_and_end_summary flow,
reusing llm_call_execute plus the current interceptors/subscriber setup, and
assert each call’s emitted marks and annotated_response().optimization_summary
only contain that call’s own contributions with no cross-request mixing.
In `@crates/core/tests/unit/atif_tests.rs`:
- Around line 1499-1560: Add test coverage around the optimization aggregation
in extract_metrics and compute_final_metrics by adding a non-USD summary case
where estimated_cost_saved_usd is not populated, and a no-optimization-data case
where compute_final_metrics over steps without optimization metrics leaves
final_metrics.extra as None. Use the existing test helpers and
AtifStep/optimization_summary setup to locate the relevant paths.
---
Outside diff comments:
In `@crates/ffi/src/api/mod.rs`:
- Around line 261-323: `nemo_relay_llm_request_intercept_outcome_json_new` is
dropping caller-provided optimization contributions by always setting
`optimization_contributions` to an empty vector. Update this constructor to
accept an additional optional JSON input for contributions, parse it alongside
`annotated_json` and `pending_marks_json`, and pass the parsed value into
`LlmRequestInterceptOutcome` so the FFI API matches the Python and Node
constructors.
In `@crates/python/src/py_types/codecs.rs`:
- Around line 511-547: Expose the missing optimization data in
PyAnnotatedLLMResponse so Python can read runtime-produced optimization details.
Update the PyAnnotatedLLMResponse wrapper around AnnotatedLLMResponse to add a
getter for optimization_summary, matching the core response type’s exposed
fields and keeping parity with the existing message/tool_calls/usage accessors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: e891b0f2-09c1-4387-bacd-47190e893a2c
📒 Files selected for processing (38)
crates/adaptive/tests/coverage/subscriber_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/core/src/api/llm.rscrates/core/src/api/mod.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/state.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/core/src/observability/atif.rscrates/core/src/observability/openinference.rscrates/core/src/observability/otel.rscrates/core/src/stream.rscrates/core/tests/integration/middleware_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/atif_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/node/src/api/mod.rscrates/node/src/callable.rscrates/plugin/src/lib.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/core.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/types/src/api/llm.rscrates/types/src/codec/mod.rscrates/types/src/codec/optimization.rscrates/types/src/codec/response.rscrates/types/tests/serialization_tests.rs
📜 Review details
🧰 Additional context used
📓 Path-based instructions (28)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/optimization.rscrates/core/src/codec/anthropic.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/core/src/codec/openai_responses.rscrates/core/src/observability/openinference.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/optimization.rscrates/core/src/codec/anthropic.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/core/src/codec/openai_responses.rscrates/core/src/observability/openinference.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/optimization.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/openai_responses.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/python/src/py_types/codecs.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/optimization.rscrates/core/src/codec/anthropic.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/core/src/codec/openai_responses.rscrates/core/src/observability/openinference.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings,
documentation, integrations, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile skills/ # Published Codex/agent skills for NeMo Relay usage patternsPrerequisites
Insta...
Files:
crates/core/src/api/mod.rscrates/types/src/codec/response.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/plugin/src/lib.rscrates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/core/src/codec/optimization.rscrates/types/src/codec/mod.rscrates/core/src/codec/anthropic.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/src/codec/openai_responses.rscrates/types/src/api/llm.rscrates/core/src/observability/openinference.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/node/src/api/mod.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/types/src/codec/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/optimization.rscrates/core/src/codec/anthropic.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/core/src/codec/openai_responses.rscrates/core/src/observability/openinference.rscrates/core/src/stream.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/src/api/shared.rscrates/core/src/api/runtime/state.rscrates/core/src/observability/otel.rscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
{crates/core/src/plugin/dynamic/**,crates/plugin/**,crates/worker/**,crates/worker-proto/**,crates/types/**,python/plugin/**,examples/rust-native-plugin/**,examples/python-grpc-worker-plugin/**,docs/build-plugins/**}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Keep the stable boundary explicit: native plugins cross a C ABI, and worker plugins cross
grpc-v1.
Files:
crates/types/src/codec/response.rscrates/plugin/src/lib.rscrates/types/src/codec/mod.rscrates/types/tests/serialization_tests.rscrates/types/src/api/llm.rscrates/types/src/codec/optimization.rs
crates/types/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Runtime helpers must cover marks, scopes, continuations, and isolated scope stacks.
Files:
crates/types/src/codec/response.rscrates/types/src/codec/mod.rscrates/types/tests/serialization_tests.rscrates/types/src/api/llm.rscrates/types/src/codec/optimization.rs
{crates/core/src/plugin/dynamic/**/*.rs,crates/plugin/**/*.rs,crates/worker/**/*.rs,crates/worker-proto/**/*.rs,python/plugin/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Manifest validation must cover kind, compatibility, load contract, integrity, capability mismatch, and disabled-plugin behavior.
Files:
crates/plugin/src/lib.rs
{crates/plugin/**/*.rs,python/plugin/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Rust and Python SDKs must expose every supported registration surface.
Files:
crates/plugin/src/lib.rs
crates/ffi/**
📄 CodeRabbit inference engine (.agents/skills/test-ffi-surface/SKILL.md)
Rebuild the FFI crate in release mode so the shared library and header stay in sync when making changes to crates/ffi
Files:
crates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rs
crates/ffi/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/ffi, also usetest-ffi-surfacefor validationUse C FFI export names prefixed with
nemo_relay_in the raw C FFI layer.
Files:
crates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rs
crates/{python,ffi,node}/**/*
⚙️ CodeRabbit configuration file
crates/{python,ffi,node}/**/*: Treat binding changes as public API changes. Check for parity with the other language bindings, FFI ownership/lifetime safety,
callback error propagation, stable type conversion, and consistent async/stream semantics.
Flag changes that update one binding without corresponding tests or documentation for the same surface elsewhere.
Files:
crates/ffi/tests/unit/types_tests.rscrates/ffi/src/api/mod.rscrates/python/src/py_types/codecs.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/python/src/py_types/core.rscrates/node/src/callable.rscrates/node/src/api/mod.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/ffi/tests/unit/types_tests.rscrates/types/tests/serialization_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/types_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/tests/unit/atif_tests.rscrates/core/tests/integration/middleware_tests.rs
{crates/ffi/src/api/*.rs,crates/ffi/nemo_relay.h}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Add or update the shared C/FFI surface in the relevant
crates/ffi/src/api/*.rsmodule, re-export it throughcrates/ffi/src/api/mod.rs, and keep the generatedcrates/ffi/nemo_relay.hheader correct.
Files:
crates/ffi/src/api/mod.rs
crates/adaptive/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep
crates/adaptivealigned with the canonical adaptive config schema, built-in section helpers, plugin lifecycle, and validation/report behavior.
Files:
crates/adaptive/tests/coverage/subscriber_tests.rscrates/adaptive/tests/unit/drain_tests.rs
crates/core/src/observability/{atif,otel,openinference}.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
When changing the core event model, emitted fields, exporter behavior, subscriber config, or lifecycle, keep the ATIF, OpenTelemetry, and OpenInference implementations in sync.
Files:
crates/core/src/observability/openinference.rscrates/core/src/observability/otel.rscrates/core/src/observability/atif.rs
{crates/python/src/py_api/mod.rs,python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go,crates/node/src/api/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update the language-native bindings for every exposed surface in Python, Go, and Node.js.
Files:
crates/node/src/api/mod.rs
crates/core/src/api/runtime/state.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
crates/core/src/api/runtime/state.rs: Add registry fields asSortedRegistry<GuardrailEntry<T>>orSortedRegistry<Intercept<T>>toNemoRelayContextStateincrates/core/src/api/runtime/state.rs
Add chain execution helpers toNemoRelayContextStatefollowing the pattern of existing methods liketool_sanitize_request_chainortool_request_intercepts_chain
Files:
crates/core/src/api/runtime/state.rs
crates/core/src/api/{tool,llm}.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Wire the new middleware chain into the execute path in
crates/core/src/api/tool.rsorcrates/core/src/api/llm.rsat the appropriate pipeline stage
Files:
crates/core/src/api/llm.rs
🔇 Additional comments (39)
crates/core/src/api/runtime/state.rs (1)
1140-1140: LGTM!Also applies to: 1159-1159, 1168-1168
crates/core/src/api/shared.rs (1)
191-196: LGTM!Also applies to: 232-250
crates/core/tests/unit/shared_tests.rs (1)
165-165: LGTM!Also applies to: 202-202, 272-272, 299-299, 336-336, 378-378
crates/core/src/api/llm.rs (4)
12-17: LGTM!Also applies to: 73-76
368-402: LGTM!
538-572: LGTM!Also applies to: 607-635
741-742: LGTM!Also applies to: 767-770, 784-789, 915-916, 941-944, 959-964
crates/core/src/stream.rs (1)
36-37: LGTM!Also applies to: 213-235
crates/types/src/codec/optimization.rs (1)
1-353: LGTM!crates/types/src/codec/mod.rs (1)
6-7: LGTM!crates/types/src/codec/response.rs (1)
13-13: LGTM!Also applies to: 54-57
crates/types/src/api/llm.rs (1)
11-11: LGTM!Also applies to: 52-54, 64-64, 74-83
crates/core/src/api/mod.rs (1)
10-11: LGTM!crates/core/src/api/optimization.rs (3)
23-39: LGTM!Also applies to: 92-129, 132-189
191-361: LGTM!
363-728: LGTM!crates/core/src/codec/optimization.rs (1)
1-6: LGTM!crates/core/src/codec/mod.rs (1)
21-21: LGTM!crates/core/tests/integration/middleware_tests.rs (1)
25-25: LGTM!Also applies to: 59-62
crates/core/tests/integration/native_plugin_tests.rs (1)
525-525: LGTM!crates/core/tests/unit/types_tests.rs (1)
126-126: LGTM!crates/ffi/tests/unit/types_tests.rs (1)
639-639: LGTM!crates/python/tests/coverage/py_types_coverage_tests.rs (1)
626-626: LGTM!Also applies to: 1221-1221, 1265-1265, 1484-1484
crates/core/tests/integration/pipeline_tests.rs (1)
1168-1188: LGTM!crates/adaptive/tests/coverage/subscriber_tests.rs (1)
128-138: LGTM!crates/adaptive/tests/unit/drain_tests.rs (1)
850-878: LGTM!Also applies to: 986-996
crates/core/tests/unit/codec/response_tests.rs (1)
39-66: LGTM!Also applies to: 69-81, 1694-1707
crates/types/tests/serialization_tests.rs (1)
44-54: LGTM!crates/node/src/api/mod.rs (1)
2958-2959: LGTM!Also applies to: 2974-2974
crates/node/src/callable.rs (1)
29-29: LGTM!Also applies to: 298-317
crates/python/src/py_types/core.rs (1)
690-753: LGTM!crates/plugin/src/lib.rs (1)
25-29: 🎯 Functional CorrectnessNo change needed for
DataSchemaLlmOptimizationContribution::with_payloadbuilds the schema internally, andDataSchemais still available fromnemo_relay_types::api::eventif manualpayload_schemaconstruction is needed.> Likely an incorrect or invalid review comment.crates/core/src/codec/anthropic.rs (1)
406-417: LGTM!crates/core/src/codec/openai_chat.rs (1)
212-223: LGTM!crates/core/src/codec/openai_responses.rs (1)
540-551: LGTM!crates/core/src/observability/atif.rs (2)
773-822: LGTM!
1230-1296: LGTM!crates/core/tests/unit/atif_tests.rs (1)
103-115: LGTM!crates/core/src/observability/openinference.rs (1)
911-914: 📐 Maintainability & Code QualityPlease confirm the required core-Rust validation matrix ran for this change.
This touches
crates/coreobservability behavior, so the PR should explicitly include the Rust checks (cargo fmt --all,cargo clippy --workspace --all-targets -- -D warnings,just test-rust,cargo deny check) and the broader affected-language validation called out for core changes.As per coding guidelines,
**/*.rs: "Any Rust change must runjust test-rust", "Any Rust change must runcargo fmt --all", and "Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings". As per coding guidelines,crates/core/**/*.rs: "If the change touchedcrates/coreor shared runtime semantics, also usevalidate-changefor broader validation". As per path instructions,crates/{core,adaptive}/**/*.rs: "Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions."Also applies to: 916-1004
Sources: Coding guidelines, Path instructions
There was a problem hiding this comment.
(Codex Assisted)
Requesting changes for runtime safety, accounting correctness, and binding parity.
I confirmed the existing unresolved threads for the incomplete contribution-size bound and the missing per-cost currency/provenance attributes; those should remain merge blockers and are not duplicated below.
Two additional binding gaps are outside the changed-file diff and therefore cannot be attached inline:
python/plugin/src/nemo_relay_plugin/_api.py: the slottedLlmRequestInterceptOutcomehas nooptimization_contributionsfield andto_json()never emits it, so Python gRPC worker plugins cannot use the new contract.python/nemo_relay/_native.pyi:LLMRequestInterceptOutcomestill advertises the old constructor and omits the new property, so typed Python callers reject the runtime-supported API.
Please add regression coverage for mark sanitization, routed-model lifecycle pricing, Python worker round-tripping, non-USD/mixed-provenance export, concurrent call isolation, and the overflow/partial-summary branches.
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
bbednarski9
force-pushed
the
bbednarski/llm-optimization-accounting
branch
from
fd3a52a to
066663c
Compare
github-actions
Bot
added
size:XXL
There was a problem hiding this comment.
Actionable comments posted: 8
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/core/src/api/llm.rs (1)
797-857: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftDuplicated optimization-recorder wiring between buffered and streaming call paths.
The recorder creation → scoped-intercept run → handle assignment → mark emission → scoped-execution wrap sequence (lines 797-857) is repeated almost verbatim in
llm_stream_call_execute(lines 983-1043). Extracting a shared helper (e.g.fn run_intercepts_and_wire_recorder(...)returning the intercepted request/annotated request/pending marks/handle) would reduce the chance the two paths drift when this logic changes again.Also applies to: 983-1043
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/core/src/api/llm.rs` around lines 797 - 857, The buffered and streaming LLM execution paths duplicate the same optimization-recorder setup and wiring logic, so factor the shared sequence in this area of llm.rs into a helper that both call sites can reuse. Extract the recorder creation, scope_llm_optimization_recorder/run_request_intercepts_with_codec_and_recorder, handle initialization via create_llm_handle, subscriber snapshotting, emit_llm_start_with_subscribers/emit_pending_request_marks/emit_optimization_marks, and the scoped execution wrapper into a shared function such as run_intercepts_and_wire_recorder, and have both the current function and llm_stream_call_execute delegate to it.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/core/src/observability/otel.rs`:
- Around line 718-859: The `push_optimization_attributes` function repeats
nearly identical logic for `baseline_cost` and `actual_cost`, which increases
drift risk. Extract the shared cost attribute emission into a helper (for
example, a cost provenance helper that takes `attributes`, a prefix, and the
cost object) and have both `baseline_cost` and `actual_cost` blocks call it.
Keep the existing `pricing_source` and `pricing_as_of` behavior intact, and use
the helper to centralize the total/currency/provenance attribute mapping.
In `@crates/core/tests/integration/pipeline_tests.rs`:
- Around line 1662-1801: The routed pricing test leaves global pricing-resolver
state behind if any assertion or unwrap panics before the final cleanup. Add a
panic-safe guard in this test around the setup done by
install_routed_response_pricing() so reset_active_pricing_resolver() runs from
Drop unconditionally, and keep the explicit cleanup only if needed; use the
test’s existing symbols like install_routed_response_pricing,
reset_active_pricing_resolver, and the new guard near the current TEST_MUTEX
setup.
In `@crates/core/tests/integration/stream_tests.rs`:
- Around line 279-423: The stream termination test has heavy duplication across
the clean, error-before, error-after, drop-before, and drop-after scenarios.
Refactor the repeated `make_optimized_llm_handle` + `make_collector_finalizer` +
`LlmStreamWrapper::new` setup in
`stream_termination_modes_close_accounting_without_losing_evidence` into a small
helper that accepts the stream items and returns the recorder and wrapper, so
each scenario only specifies its differing inputs.
In `@crates/node/src/api/mod.rs`:
- Line 2266: The inline TypeScript signatures in the api module have drifted
from the canonical LlmOptimizationContribution definition in plugin.d.ts because
they still type kind as a plain string. Update the ts_arg_type/ts_return_type
strings used in this module so kind matches the public literal-union type
('input_compression' | 'model_routing' | (string & {})), and prefer sharing the
type text through a single reusable constant or helper to keep the copies in
sync across the affected signatures.
In `@crates/python/src/py_types/codecs.rs`:
- Around line 621-636: The `optimization_summary` getter in `codecs.rs` should
follow the same shared JSON conversion path as the sibling getters (`message`,
`tool_calls`, `usage`, and `api_specific`) instead of calling
`serde_json::to_value` and `json_to_py` directly. Update `optimization_summary`
to use `to_python_json_value(...)` with the existing
`FORCE_ANNOTATED_RESPONSE_*_SERIALIZATION_ERROR` test hook pattern so its
serialization failure path is covered consistently. Make sure the returned
Python value and error handling stay aligned with the other JSON getters in
`PyAnnotatedResponse`.
In `@crates/worker/tests/optimization_contract.rs`:
- Around line 23-28: The `optimization_contract` test is using weak negative
checks on `contribution.kind`, so it should assert the decoded kind matches the
value from the fixture instead of only not being `input_compression()` or
`model_routing()`. Update the assertions in `optimization_contract.rs` to
compare `contribution.kind` against the expected fixture kind (using the same
string/value that the fixture encodes, via `as_str()` or equivalent) alongside
the existing `extra["future_top_level_field"]` round-trip check.
In `@go/nemo_relay/optimization.go`:
- Around line 109-172: The Go JSON binding for LLMOptimizationContribution is
missing the payload/payload_schema invariant enforced by the Python/Rust path.
Add the same validation in LLMOptimizationContribution.UnmarshalJSON and, if
needed, MarshalJSON so payload cannot be present when PayloadSchema is empty,
and return a local error before the value can round-trip. Use the existing
llmOptimizationContributionWire decoding and the LLMOptimizationContribution
struct fields to enforce parity with the core recorder behavior.
In `@python/plugin/src/nemo_relay_plugin/_api.py`:
- Around line 411-424: The default for LlmOptimizationContribution.applied is
inconsistent between direct instantiation and
LlmOptimizationContribution.from_json, so align both paths to treat an omitted
applied value the same way. Update the dataclass default and the decoding logic
in from_json together so the same default is used when the field is missing, and
verify any related parsing helpers in the same class keep that behavior
consistent.
---
Outside diff comments:
In `@crates/core/src/api/llm.rs`:
- Around line 797-857: The buffered and streaming LLM execution paths duplicate
the same optimization-recorder setup and wiring logic, so factor the shared
sequence in this area of llm.rs into a helper that both call sites can reuse.
Extract the recorder creation,
scope_llm_optimization_recorder/run_request_intercepts_with_codec_and_recorder,
handle initialization via create_llm_handle, subscriber snapshotting,
emit_llm_start_with_subscribers/emit_pending_request_marks/emit_optimization_marks,
and the scoped execution wrapper into a shared function such as
run_intercepts_and_wire_recorder, and have both the current function and
llm_stream_call_execute delegate to it.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 0e6e5af9-fd32-4f51-87e7-510f28be6a4d
📒 Files selected for processing (59)
crates/adaptive/tests/coverage/subscriber_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/core/src/api/llm.rscrates/core/src/api/mod.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/core/src/observability/atif.rscrates/core/src/observability/openinference.rscrates/core/src/observability/otel.rscrates/core/src/stream.rscrates/core/tests/integration/middleware_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/integration/stream_tests.rscrates/core/tests/unit/atif_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/tests/unit/stream_tests.rscrates/core/tests/unit/types_tests.rscrates/ffi/nemo_relay.hcrates/ffi/src/api/mod.rscrates/ffi/tests/unit/api/core_tests.rscrates/ffi/tests/unit/types_tests.rscrates/node/plugin.d.tscrates/node/src/api/mod.rscrates/node/src/callable.rscrates/node/tests/llm_tests.mjscrates/plugin/src/lib.rscrates/plugin/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/core.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/types/src/api/llm.rscrates/types/src/codec/mod.rscrates/types/src/codec/optimization.rscrates/types/src/codec/response.rscrates/types/tests/fixtures/llm_optimization_contribution_v1.jsoncrates/types/tests/optimization_tests.rscrates/types/tests/serialization_tests.rscrates/worker/src/lib.rscrates/worker/tests/optimization_contract.rsgo/nemo_relay/callbacks.gogo/nemo_relay/optimization.gogo/nemo_relay/optimization_test.gopython/nemo_relay/_native.pyipython/plugin/src/nemo_relay_plugin/__init__.pypython/plugin/src/nemo_relay_plugin/_api.pypython/tests/plugin/test_worker_sdk.py
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (43)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/core/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rscrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/core/tests/unit/stream_tests.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rscrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/atif_tests.rscrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/adaptive/tests/unit/drain_tests.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/stream_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/atif_tests.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/core/tests/unit/codec/response_tests.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/stream_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/atif_tests.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/core/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/core/tests/unit/stream_tests.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rscrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/atif_tests.rscrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rscrates/core/tests/unit/observability/otel_tests.rspython/plugin/src/nemo_relay_plugin/_api.pycrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/core/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/core/tests/unit/stream_tests.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rscrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/node/tests/llm_tests.mjscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/tests/integration/native_plugin_tests.rscrates/node/plugin.d.tscrates/core/tests/unit/atif_tests.rscrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rscrates/core/tests/unit/observability/otel_tests.rspython/plugin/src/nemo_relay_plugin/_api.pycrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/core/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/ffi/nemo_relay.hcrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/core/tests/unit/stream_tests.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rsgo/nemo_relay/optimization_test.gocrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rsgo/nemo_relay/callbacks.gocrates/core/src/codec/anthropic.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/tests/integration/native_plugin_tests.rscrates/node/plugin.d.tscrates/core/tests/unit/atif_tests.rscrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rscrates/core/tests/unit/observability/otel_tests.rsgo/nemo_relay/optimization.gopython/plugin/src/nemo_relay_plugin/_api.pycrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/core/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rscrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/core/tests/unit/stream_tests.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rsgo/nemo_relay/optimization_test.gocrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rsgo/nemo_relay/callbacks.gocrates/core/src/codec/anthropic.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/node/plugin.d.tscrates/core/tests/unit/atif_tests.rscrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rscrates/core/tests/unit/observability/otel_tests.rsgo/nemo_relay/optimization.gocrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/core/src/codec/openai_chat.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/api/runtime/state.rscrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rspython/tests/plugin/test_worker_sdk.pycrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rspython/plugin/src/nemo_relay_plugin/_api.pycrates/core/src/api/optimization.rscrates/core/src/api/llm.rscrates/core/src/observability/otel.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/core/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/types/tests/fixtures/llm_optimization_contribution_v1.jsoncrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/ffi/nemo_relay.hcrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/core/tests/unit/stream_tests.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rsgo/nemo_relay/optimization_test.gocrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/node/tests/llm_tests.mjscrates/core/src/api/shared.rsgo/nemo_relay/callbacks.gopython/nemo_relay/_native.pyicrates/core/src/codec/anthropic.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/tests/integration/native_plugin_tests.rscrates/node/plugin.d.tscrates/core/tests/unit/atif_tests.rscrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rscrates/core/tests/unit/observability/otel_tests.rsgo/nemo_relay/optimization.gopython/plugin/src/nemo_relay_plugin/_api.pycrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/adaptive/tests/unit/drain_tests.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/stream_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/atif_tests.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/core/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/core/tests/unit/stream_tests.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rsgo/nemo_relay/optimization_test.gocrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rsgo/nemo_relay/callbacks.gocrates/core/src/codec/anthropic.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/tests/integration/native_plugin_tests.rscrates/node/plugin.d.tscrates/core/tests/unit/atif_tests.rscrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rscrates/core/tests/unit/observability/otel_tests.rsgo/nemo_relay/optimization.gopython/plugin/src/nemo_relay_plugin/_api.pycrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings,
documentation, integrations, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile skills/ # Published Codex/agent skills for NeMo Relay usage patternsPrerequisites
Insta...
Files:
crates/core/src/api/mod.rscrates/ffi/tests/unit/types_tests.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/types/tests/fixtures/llm_optimization_contribution_v1.jsoncrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/node/src/callable.rscrates/core/src/codec/openai_chat.rscrates/ffi/nemo_relay.hcrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/core/tests/unit/stream_tests.rscrates/plugin/src/lib.rscrates/core/src/observability/openinference.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rsgo/nemo_relay/optimization_test.gocrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/node/tests/llm_tests.mjscrates/core/src/api/shared.rsgo/nemo_relay/callbacks.gopython/nemo_relay/_native.pyicrates/core/src/codec/anthropic.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/tests/integration/native_plugin_tests.rscrates/node/plugin.d.tscrates/core/tests/unit/atif_tests.rscrates/worker/src/lib.rscrates/node/src/api/mod.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/ffi/src/api/mod.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/types/src/codec/optimization.rscrates/core/tests/unit/observability/otel_tests.rsgo/nemo_relay/optimization.gopython/plugin/src/nemo_relay_plugin/_api.pycrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/src/api/mod.rscrates/core/src/codec/mod.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/core/tests/unit/codec/response_tests.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/optimization.rscrates/adaptive/tests/unit/drain_tests.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/stream_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/types_tests.rscrates/core/src/api/shared.rscrates/core/src/codec/anthropic.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/atif_tests.rscrates/core/src/stream.rscrates/core/src/observability/atif.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/tests/integration/stream_tests.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/llm.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/observability/otel.rs
crates/ffi/**
📄 CodeRabbit inference engine (.agents/skills/test-ffi-surface/SKILL.md)
Rebuild the FFI crate in release mode so the shared library and header stay in sync when making changes to crates/ffi
Files:
crates/ffi/tests/unit/types_tests.rscrates/ffi/tests/unit/api/core_tests.rscrates/ffi/nemo_relay.hcrates/ffi/src/api/mod.rs
crates/ffi/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/ffi, also usetest-ffi-surfacefor validationUse C FFI export names prefixed with
nemo_relay_in the raw C FFI layer.
Files:
crates/ffi/tests/unit/types_tests.rscrates/ffi/tests/unit/api/core_tests.rscrates/ffi/src/api/mod.rs
crates/{python,ffi,node}/**/*
⚙️ CodeRabbit configuration file
crates/{python,ffi,node}/**/*: Treat binding changes as public API changes. Check for parity with the other language bindings, FFI ownership/lifetime safety,
callback error propagation, stable type conversion, and consistent async/stream semantics.
Flag changes that update one binding without corresponding tests or documentation for the same surface elsewhere.
Files:
crates/ffi/tests/unit/types_tests.rscrates/ffi/tests/unit/api/core_tests.rscrates/node/src/callable.rscrates/ffi/nemo_relay.hcrates/python/src/py_types/core.rscrates/python/src/py_types/codecs.rscrates/node/tests/llm_tests.mjscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/plugin.d.tscrates/node/src/api/mod.rscrates/ffi/src/api/mod.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/ffi/tests/unit/types_tests.rscrates/adaptive/tests/coverage/subscriber_tests.rscrates/worker/tests/optimization_contract.rscrates/core/tests/unit/codec/response_tests.rscrates/types/tests/serialization_tests.rscrates/types/tests/fixtures/llm_optimization_contribution_v1.jsoncrates/ffi/tests/unit/api/core_tests.rscrates/adaptive/tests/unit/drain_tests.rscrates/types/tests/optimization_tests.rscrates/core/tests/unit/stream_tests.rsgo/nemo_relay/optimization_test.gocrates/plugin/tests/optimization_contract.rscrates/core/tests/unit/types_tests.rscrates/node/tests/llm_tests.mjscrates/python/tests/coverage/py_types_coverage_tests.rscrates/core/tests/unit/observability/openinference_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/tests/integration/native_plugin_tests.rscrates/core/tests/unit/atif_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/shared_tests.rscrates/core/tests/integration/stream_tests.rscrates/core/tests/unit/observability/otel_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/core/tests/unit/optimization_tests.rscrates/core/tests/integration/middleware_tests.rs
crates/adaptive/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep
crates/adaptivealigned with the canonical adaptive config schema, built-in section helpers, plugin lifecycle, and validation/report behavior.
Files:
crates/adaptive/tests/coverage/subscriber_tests.rscrates/adaptive/tests/unit/drain_tests.rs
{crates/core/src/plugin/dynamic/**,crates/plugin/**,crates/worker/**,crates/worker-proto/**,crates/types/**,python/plugin/**,examples/rust-native-plugin/**,examples/python-grpc-worker-plugin/**,docs/build-plugins/**}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Keep the stable boundary explicit: native plugins cross a C ABI, and worker plugins cross
grpc-v1.
Files:
crates/worker/tests/optimization_contract.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/types/tests/fixtures/llm_optimization_contribution_v1.jsoncrates/types/tests/optimization_tests.rscrates/plugin/src/lib.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rscrates/plugin/tests/optimization_contract.rscrates/worker/src/lib.rscrates/types/src/codec/optimization.rspython/plugin/src/nemo_relay_plugin/_api.py
{crates/worker-proto/**/*.{rs,proto},crates/worker/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Keep worker protocol DTOs in
JsonEnvelope; protobuf should own control flow, not duplicated Relay data models.
Files:
crates/worker/tests/optimization_contract.rscrates/worker/src/lib.rs
{crates/core/src/plugin/dynamic/**/*.rs,crates/plugin/**/*.rs,crates/worker/**/*.rs,crates/worker-proto/**/*.rs,python/plugin/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Manifest validation must cover kind, compatibility, load contract, integrity, capability mismatch, and disabled-plugin behavior.
Files:
crates/worker/tests/optimization_contract.rspython/plugin/src/nemo_relay_plugin/__init__.pycrates/plugin/src/lib.rscrates/plugin/tests/optimization_contract.rscrates/worker/src/lib.rspython/plugin/src/nemo_relay_plugin/_api.py
crates/worker/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Worker activation must cover process launch, token authentication, handshake, validation, declarative registration, proxy rollback, cancellation, and shutdown.
Files:
crates/worker/tests/optimization_contract.rscrates/worker/src/lib.rs
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: When changing the Python wrapper package, tests, or docs tooling, lint with Ruff (E,F,W,I), format with Ruff formatter (120-character lines, double quotes), and passtytype checking.
Add the SPDX license header to all Python source files using the#comment form.
Files:
python/plugin/src/nemo_relay_plugin/__init__.pypython/tests/plugin/test_worker_sdk.pypython/plugin/src/nemo_relay_plugin/_api.py
{crates/plugin/**/*.rs,python/plugin/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Rust and Python SDKs must expose every supported registration surface.
Files:
python/plugin/src/nemo_relay_plugin/__init__.pycrates/plugin/src/lib.rscrates/plugin/tests/optimization_contract.rspython/plugin/src/nemo_relay_plugin/_api.py
**/*.{py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Keep Python, Go, and Node.js config objects and subscriber/exporter methods aligned so all bindings expose the same logical knobs and semantics.
Files:
python/plugin/src/nemo_relay_plugin/__init__.pygo/nemo_relay/optimization_test.gogo/nemo_relay/callbacks.gopython/tests/plugin/test_worker_sdk.pycrates/node/plugin.d.tsgo/nemo_relay/optimization.gopython/plugin/src/nemo_relay_plugin/_api.py
crates/types/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Runtime helpers must cover marks, scopes, continuations, and isolated scope stacks.
Files:
crates/types/tests/serialization_tests.rscrates/types/src/codec/mod.rscrates/types/tests/optimization_tests.rscrates/types/src/codec/response.rscrates/types/src/api/llm.rscrates/types/src/codec/optimization.rs
crates/ffi/nemo_relay.h
📄 CodeRabbit inference engine (.agents/skills/test-ffi-surface/SKILL.md)
Check the generated header diff when any exported symbol or type changed in the FFI surface
Update generated or generated-from-build surfaces such as
crates/ffi/nemo_relay.hthrough the proper build step.
Files:
crates/ffi/nemo_relay.h
{crates/ffi/src/api/*.rs,crates/ffi/nemo_relay.h}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Add or update the shared C/FFI surface in the relevant
crates/ffi/src/api/*.rsmodule, re-export it throughcrates/ffi/src/api/mod.rs, and keep the generatedcrates/ffi/nemo_relay.hheader correct.
Files:
crates/ffi/nemo_relay.hcrates/ffi/src/api/mod.rs
crates/core/src/observability/{atif,otel,openinference}.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
When changing the core event model, emitted fields, exporter behavior, subscriber config, or lifecycle, keep the ATIF, OpenTelemetry, and OpenInference implementations in sync.
Files:
crates/core/src/observability/openinference.rscrates/core/src/observability/atif.rscrates/core/src/observability/otel.rs
go/nemo_relay/**/*.go
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
go/nemo_relay/**/*.go: Format changed Go packages withcd go/nemo_relay && go fmt ./...
Run Go tests withjust test-goto build and test the NeMo Relay Go binding
Usejust build-gowhen you want an explicit build-only pass or need the artifact for other work
Usejust ci=true test-gowhen you need the CI-style coverage and JUnit path
On macOS, setDYLD_LIBRARY_PATHto the../../target/releasedirectory before running the rawgo testcommand directlyUse
PascalCasefor public Go APIs.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/callbacks.gogo/nemo_relay/optimization.go
**/*.go
📄 CodeRabbit inference engine (CONTRIBUTING.md)
When changing the experimental Go binding, format Go code with
gofmtand keepgo vet ./...passing.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/callbacks.gogo/nemo_relay/optimization.go
{crates/python/src/py_api/mod.rs,python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go,crates/node/src/api/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update the language-native bindings for every exposed surface in Python, Go, and Node.js.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/callbacks.gopython/nemo_relay/_native.pyicrates/node/src/api/mod.rsgo/nemo_relay/optimization.go
{python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update language wrapper helpers such as Python wrapper modules, Python type stubs, and Go shorthand packages when the new behavior belongs in those helper layers.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/callbacks.gopython/nemo_relay/_native.pyigo/nemo_relay/optimization.go
go/nemo_relay/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep shared plugin helpers in
go/nemo_relayaligned with plugin registration, composition, and lifecycle behavior.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/callbacks.gogo/nemo_relay/optimization.go
go/nemo_relay/**/*
⚙️ CodeRabbit configuration file
go/nemo_relay/**/*: Review Go binding changes for cgo memory ownership, race safety, callback cleanup, idiomatic exported APIs, and parity with Rust/FFI behavior.
Any API change should include focused Go tests and consider race-test behavior.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/callbacks.gogo/nemo_relay/optimization.go
crates/node/**/*.{js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
camelCasefor Node.js public APIs.
Files:
crates/node/tests/llm_tests.mjscrates/node/plugin.d.ts
python/nemo_relay/**/*
⚙️ CodeRabbit configuration file
python/nemo_relay/**/*: Review Python wrapper changes for typed API consistency, contextvars-based scope isolation, async behavior, and parity with the native extension.
Stubs and runtime implementations should stay aligned.
Files:
python/nemo_relay/_native.pyi
python/tests/**/*.py
📄 CodeRabbit inference engine (.agents/skills/test-python-binding/SKILL.md)
python/tests/**/*.py: Pytest is used to run tests.
Do not add@pytest.mark.asyncioto any test; async tests are automatically detected and run by the async runner.
Do not add a-> Nonereturn type annotation to test functions.
When mocking a class, do not define a new class; useunittest.mock.MagicMockorunittest.mock.AsyncMock, with thespecconstructor argument when necessary.
Name mocked classes with themockprefix, notfake.
Prefer pytest fixtures over helper methods.
Do not repeat fixtures; if a fixture is needed in multiple test files, place it in aconftest.pyfile.
When creating a fixture, use@pytest.fixture(name="<fixture_name>"[, scope="<scope>"])and define the fixture function asdef <fixture_name>_fixture() -> <return_type>:; only specifyscopewhen it is notfunction.
Preferpytest.mark.parametrizeover creating individual tests for different input types.
Files:
python/tests/plugin/test_worker_sdk.py
crates/node/**/*.{js,ts,jsx,tsx,json}
📄 CodeRabbit inference engine (.agents/skills/test-node-binding/SKILL.md)
Format changed Node files with
npm run format --workspace=nemo-relay-node
Files:
crates/node/plugin.d.ts
crates/node/**/*.{ts,tsx,d.ts}
📄 CodeRabbit inference engine (.agents/skills/test-node-binding/SKILL.md)
Use
npm run check:docstrings --workspace=nemo-relay-nodeto validate public API docstring checks when surface docs changed
Files:
crates/node/plugin.d.ts
crates/core/src/api/runtime/state.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
crates/core/src/api/runtime/state.rs: Add registry fields asSortedRegistry<GuardrailEntry<T>>orSortedRegistry<Intercept<T>>toNemoRelayContextStateincrates/core/src/api/runtime/state.rs
Add chain execution helpers toNemoRelayContextStatefollowing the pattern of existing methods liketool_sanitize_request_chainortool_request_intercepts_chain
Files:
crates/core/src/api/runtime/state.rs
crates/core/src/api/{tool,llm}.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Wire the new middleware chain into the execute path in
crates/core/src/api/tool.rsorcrates/core/src/api/llm.rsat the appropriate pipeline stage
Files:
crates/core/src/api/llm.rs
🧠 Learnings (1)
📚 Learning: 2026-05-07T18:04:44.387Z
Learnt from: mnajafian-nv
Repo: NVIDIA/NeMo-Flow PR: 67
File: integrations/openclaw/src/modules.ts:1-2
Timestamp: 2026-05-07T18:04:44.387Z
Learning: In NVIDIA/NeMo-Flow, TypeScript source files should use `//` line comments for SPDX headers (e.g., `// SPDX-FileCopyrightText: ...` and `// SPDX-License-Identifier: ...`) rather than C-style block comments (`/* ... */`). The repo’s copyright checker enforces this mapping, so `//` SPDX headers in `.ts` files should not be flagged as a style violation.
Applied to files:
crates/node/plugin.d.ts
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
bbednarski9
force-pushed
the
bbednarski/llm-optimization-accounting
branch
from
3a804da to
ded7a34
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/core/src/api/optimization.rs (1)
32-43: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winConsider collapsing the parallel
contributions/recorded_atvectors into one.Correct as written —
recorded_atis pushed/cleared in lockstep withcontributionsat every mutation site shown (record,finish). Butunemitted_with_timestampsslices both vectors by the samestartindex and zips them, which silently assumes the invariant holds everywhere; any future edit that pushes/clears one vector without the other panics on an out-of-bounds slice instead of failing a type check. StoringVec<(LlmOptimizationContribution, DateTime<Utc>)>(or a small struct) instead of two syncedVecs removes this manually-maintained invariant entirely.♻️ Suggested consolidation
struct AccumulatorState { - contributions: Vec<LlmOptimizationContribution>, - recorded_at: Vec<DateTime<Utc>>, + contributions: Vec<(LlmOptimizationContribution, DateTime<Utc>)>, total_contribution_bytes: usize, ... }Then update
record()'s push,finish()'s clear/take, andunemitted_with_timestamps()'s slice accordingly.Also applies to: 148-148, 180-205, 275-275
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/core/src/api/optimization.rs` around lines 32 - 43, The AccumulatorState in optimization.rs keeps contributions and recorded_at as two manually synchronized vectors, which is fragile and can break if future mutations diverge. Collapse them into a single collection like Vec<(LlmOptimizationContribution, DateTime<Utc>)> or a small paired struct, then update record() to push the pair, finish() to clear/take the paired entries, and unemitted_with_timestamps() to iterate or slice the combined items instead of zipping parallel vectors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/plugin/src/nemo_relay_plugin/_api.py`:
- Line 477: Align the omitted applied default across the binding implementations
for LlmOptimizationContribution/from_json so the same JSON means the same thing
everywhere. Decide whether the contract should default missing applied to true
or false, then update the Python fallback in from_json together with the Rust
serde default and the Go decode behavior to match, using the relevant
LlmOptimizationContribution serialization/deserialization paths.
---
Outside diff comments:
In `@crates/core/src/api/optimization.rs`:
- Around line 32-43: The AccumulatorState in optimization.rs keeps contributions
and recorded_at as two manually synchronized vectors, which is fragile and can
break if future mutations diverge. Collapse them into a single collection like
Vec<(LlmOptimizationContribution, DateTime<Utc>)> or a small paired struct, then
update record() to push the pair, finish() to clear/take the paired entries, and
unemitted_with_timestamps() to iterate or slice the combined items instead of
zipping parallel vectors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: fb03b3fc-7513-473d-8cad-3140b545e436
📒 Files selected for processing (14)
crates/core/src/api/llm.rscrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/tests/integration/pipeline_tests.rscrates/node/src/api/mod.rscrates/node/tests/llm_tests.mjscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/worker/tests/optimization_contract.rsgo/nemo_relay/optimization.gogo/nemo_relay/optimization_test.gopython/plugin/src/nemo_relay_plugin/_api.pypython/tests/plugin/test_worker_sdk.py
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (32)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/worker/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rscrates/core/tests/integration/pipeline_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/worker/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rspython/plugin/src/nemo_relay_plugin/_api.pycrates/core/tests/integration/pipeline_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/worker/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rspython/plugin/src/nemo_relay_plugin/_api.pycrates/node/tests/llm_tests.mjscrates/core/tests/integration/pipeline_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/worker/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rsgo/nemo_relay/optimization_test.gopython/plugin/src/nemo_relay_plugin/_api.pycrates/core/tests/integration/pipeline_tests.rsgo/nemo_relay/optimization.gopython/tests/plugin/test_worker_sdk.pycrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/worker/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rsgo/nemo_relay/optimization_test.gocrates/core/tests/integration/pipeline_tests.rsgo/nemo_relay/optimization.gocrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
{crates/core/src/plugin/dynamic/**,crates/plugin/**,crates/worker/**,crates/worker-proto/**,crates/types/**,python/plugin/**,examples/rust-native-plugin/**,examples/python-grpc-worker-plugin/**,docs/build-plugins/**}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Keep the stable boundary explicit: native plugins cross a C ABI, and worker plugins cross
grpc-v1.
Files:
crates/worker/tests/optimization_contract.rspython/plugin/src/nemo_relay_plugin/_api.py
{crates/worker-proto/**/*.{rs,proto},crates/worker/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Keep worker protocol DTOs in
JsonEnvelope; protobuf should own control flow, not duplicated Relay data models.
Files:
crates/worker/tests/optimization_contract.rs
{crates/core/src/plugin/dynamic/**/*.rs,crates/plugin/**/*.rs,crates/worker/**/*.rs,crates/worker-proto/**/*.rs,python/plugin/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Manifest validation must cover kind, compatibility, load contract, integrity, capability mismatch, and disabled-plugin behavior.
Files:
crates/worker/tests/optimization_contract.rspython/plugin/src/nemo_relay_plugin/_api.py
crates/worker/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Worker activation must cover process launch, token authentication, handshake, validation, declarative registration, proxy rollback, cancellation, and shutdown.
Files:
crates/worker/tests/optimization_contract.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/worker/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rsgo/nemo_relay/optimization_test.gopython/plugin/src/nemo_relay_plugin/_api.pycrates/node/tests/llm_tests.mjscrates/core/tests/integration/pipeline_tests.rsgo/nemo_relay/optimization.gopython/tests/plugin/test_worker_sdk.pycrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/worker/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rsgo/nemo_relay/optimization_test.gopython/plugin/src/nemo_relay_plugin/_api.pycrates/core/tests/integration/pipeline_tests.rsgo/nemo_relay/optimization.gopython/tests/plugin/test_worker_sdk.pycrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings,
documentation, integrations, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile skills/ # Published Codex/agent skills for NeMo Relay usage patternsPrerequisites
Insta...
Files:
crates/worker/tests/optimization_contract.rscrates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rsgo/nemo_relay/optimization_test.gopython/plugin/src/nemo_relay_plugin/_api.pycrates/node/tests/llm_tests.mjscrates/core/tests/integration/pipeline_tests.rsgo/nemo_relay/optimization.gopython/tests/plugin/test_worker_sdk.pycrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/worker/tests/optimization_contract.rscrates/python/tests/coverage/py_types_coverage_tests.rsgo/nemo_relay/optimization_test.gocrates/node/tests/llm_tests.mjscrates/core/tests/integration/pipeline_tests.rspython/tests/plugin/test_worker_sdk.pycrates/core/tests/integration/middleware_tests.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/node/src/api/mod.rspython/plugin/src/nemo_relay_plugin/_api.pypython/tests/plugin/test_worker_sdk.pycrates/core/src/api/optimization.rscrates/core/src/api/llm.rs
crates/{python,ffi,node}/**/*
⚙️ CodeRabbit configuration file
crates/{python,ffi,node}/**/*: Treat binding changes as public API changes. Check for parity with the other language bindings, FFI ownership/lifetime safety,
callback error propagation, stable type conversion, and consistent async/stream semantics.
Flag changes that update one binding without corresponding tests or documentation for the same surface elsewhere.
Files:
crates/python/src/py_types/codecs.rscrates/python/src/py_types/mod.rscrates/python/tests/coverage/py_types_coverage_tests.rscrates/node/src/api/mod.rscrates/node/tests/llm_tests.mjs
{crates/python/src/py_api/mod.rs,python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go,crates/node/src/api/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update the language-native bindings for every exposed surface in Python, Go, and Node.js.
Files:
crates/node/src/api/mod.rsgo/nemo_relay/optimization_test.gogo/nemo_relay/optimization.go
go/nemo_relay/**/*.go
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
go/nemo_relay/**/*.go: Format changed Go packages withcd go/nemo_relay && go fmt ./...
Run Go tests withjust test-goto build and test the NeMo Relay Go binding
Usejust build-gowhen you want an explicit build-only pass or need the artifact for other work
Usejust ci=true test-gowhen you need the CI-style coverage and JUnit path
On macOS, setDYLD_LIBRARY_PATHto the../../target/releasedirectory before running the rawgo testcommand directlyUse
PascalCasefor public Go APIs.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/optimization.go
**/*.go
📄 CodeRabbit inference engine (CONTRIBUTING.md)
When changing the experimental Go binding, format Go code with
gofmtand keepgo vet ./...passing.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/optimization.go
{python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update language wrapper helpers such as Python wrapper modules, Python type stubs, and Go shorthand packages when the new behavior belongs in those helper layers.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/optimization.go
**/*.{py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Keep Python, Go, and Node.js config objects and subscriber/exporter methods aligned so all bindings expose the same logical knobs and semantics.
Files:
go/nemo_relay/optimization_test.gopython/plugin/src/nemo_relay_plugin/_api.pygo/nemo_relay/optimization.gopython/tests/plugin/test_worker_sdk.py
go/nemo_relay/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep shared plugin helpers in
go/nemo_relayaligned with plugin registration, composition, and lifecycle behavior.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/optimization.go
go/nemo_relay/**/*
⚙️ CodeRabbit configuration file
go/nemo_relay/**/*: Review Go binding changes for cgo memory ownership, race safety, callback cleanup, idiomatic exported APIs, and parity with Rust/FFI behavior.
Any API change should include focused Go tests and consider race-test behavior.
Files:
go/nemo_relay/optimization_test.gogo/nemo_relay/optimization.go
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: When changing the Python wrapper package, tests, or docs tooling, lint with Ruff (E,F,W,I), format with Ruff formatter (120-character lines, double quotes), and passtytype checking.
Add the SPDX license header to all Python source files using the#comment form.
Files:
python/plugin/src/nemo_relay_plugin/_api.pypython/tests/plugin/test_worker_sdk.py
{crates/plugin/**/*.rs,python/plugin/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Rust and Python SDKs must expose every supported registration surface.
Files:
python/plugin/src/nemo_relay_plugin/_api.py
crates/node/**/*.{js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
camelCasefor Node.js public APIs.
Files:
crates/node/tests/llm_tests.mjs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/tests/integration/pipeline_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/tests/integration/pipeline_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/tests/integration/pipeline_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/tests/integration/pipeline_tests.rscrates/core/src/api/optimization.rscrates/core/tests/integration/middleware_tests.rscrates/core/src/api/llm.rs
python/tests/**/*.py
📄 CodeRabbit inference engine (.agents/skills/test-python-binding/SKILL.md)
python/tests/**/*.py: Pytest is used to run tests.
Do not add@pytest.mark.asyncioto any test; async tests are automatically detected and run by the async runner.
Do not add a-> Nonereturn type annotation to test functions.
When mocking a class, do not define a new class; useunittest.mock.MagicMockorunittest.mock.AsyncMock, with thespecconstructor argument when necessary.
Name mocked classes with themockprefix, notfake.
Prefer pytest fixtures over helper methods.
Do not repeat fixtures; if a fixture is needed in multiple test files, place it in aconftest.pyfile.
When creating a fixture, use@pytest.fixture(name="<fixture_name>"[, scope="<scope>"])and define the fixture function asdef <fixture_name>_fixture() -> <return_type>:; only specifyscopewhen it is notfunction.
Preferpytest.mark.parametrizeover creating individual tests for different input types.
Files:
python/tests/plugin/test_worker_sdk.py
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/api/optimization.rscrates/core/src/api/llm.rs
crates/core/src/api/{tool,llm}.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Wire the new middleware chain into the execute path in
crates/core/src/api/tool.rsorcrates/core/src/api/llm.rsat the appropriate pipeline stage
Files:
crates/core/src/api/llm.rs
🔇 Additional comments (14)
crates/worker/tests/optimization_contract.rs (1)
14-49: LGTM!crates/node/src/api/mod.rs (1)
2266-2266: LGTM!Also applies to: 2742-2742, 2958-2958
python/plugin/src/nemo_relay_plugin/_api.py (2)
394-398: LGTM!
432-457: LGTM!Also applies to: 530-542
crates/core/tests/integration/pipeline_tests.rs (1)
55-61: LGTM!Also applies to: 1293-1293, 1608-1608, 1671-1808
go/nemo_relay/optimization.go (1)
109-144: LGTM!Also applies to: 147-184
python/tests/plugin/test_worker_sdk.py (1)
68-99: LGTM!Also applies to: 111-156, 1305-1341
crates/core/tests/integration/middleware_tests.rs (1)
3433-3591: LGTM!Also applies to: 3593-3678
crates/python/src/py_types/codecs.rs (1)
622-637: LGTM!
optimization_summarynow follows the sameto_python_json_value(...)+FORCE_ANNOTATED_RESPONSE_OPTIMIZATION_SUMMARY_SERIALIZATION_ERRORpattern asmessage/tool_calls/usage/api_specific, resolving the prior nitpick.crates/python/src/py_types/mod.rs (1)
56-57: LGTM!crates/python/tests/coverage/py_types_coverage_tests.rs (1)
1533-1543: LGTM!Forced-serialization coverage for
optimization_summarymirrors the sibling cases correctly.Also applies to: 1602-1606
go/nemo_relay/optimization_test.go (1)
9-9: LGTM!
TestLLMOptimizationContributionRequiresPayloadSchemacorrectly validates both marshal- and unmarshal-timepayload_schemaenforcement, plus thepayload: nullpass-through, in parity with the Rust-side invariant.Also applies to: 62-86
crates/node/tests/llm_tests.mjs (1)
871-877: LGTM!crates/core/src/api/llm.rs (1)
411-424: LGTM!Sequence-derived floor timestamp (
request_ordered_timestamp) combined withrecorded_at.max(...)correctly guarantees monotonic, sequence-consistent mark ordering even under clock/recording races — consistent with the "align mark chronology" fix.
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
|
/merge |
2 participants
Overview
This adds a plugin-neutral Relay contract for recording arbitrary LLM optimizations, retaining their raw token effects, and calculating a combined baseline-versus-actual cost estimate when the managed LLM call closes.
Summary
LlmOptimizationKind, schema-tagged custom payloads, shared model transitions, and explicit baseline/effective/saved token counts.nemo_relay.llm.optimizationmarks parented to the managed LLM handle while preserving plugin-specific marks.Runtime behavior
The close path performs no network, file, refresh, subprocess, exporter-flush, or async work. It uses an
Arcsnapshot of the active in-memory pricing resolver and a short-held non-async mutex. Work is bounded to 64 attempted/accepted contributions, 16 KiB per complete serialized contribution envelope, and 256 KiB total retained contribution data per call. A breached bound seals accounting for that call, retains the accepted prefix, produces a typed partial summary, and never changes a provider result.Token counts remain first-class evidence. Downstream systems can reprice stored summaries from baseline/effective models, observed usage, explicit token savings, and pricing provenance. Checked arithmetic exposes overflow, missing observed fields, and inconsistent totals as limitations rather than clamping or fabricating evidence. Missing usage, model identity, pricing, calculable cost totals, or currency agreement produces a partial summary with null unavailable cost fields.
Exactly one applied
model_routingcontribution may define the counterfactual baseline. Relay prefers that contribution's effective model because it is the exact target committed by the execution interceptor; provider response aliases and the requested model are fallbacks only when no applied routing target exists. Multiple applied routing contributions producemultiple_routing_contributionsand are excluded from route arithmetic while all original evidence remains visible. Non-applied decisions remain visible but have no accounting authority. When there is no routing contribution, the terminal effective model is also the baseline so compression-only savings remain calculable.Canonical optimization marks run through the mark-sanitizer chain captured with the managed LLM scope, including streaming close/drop paths. The summary embedded in the LLM end event independently runs through Relay's lifecycle sanitizer policy. Sanitizer or subscriber-dispatch failures remain best effort, retain unacknowledged mark state through the next lifecycle boundary, and never alter the LLM result.
Where should the reviewer start?
crates/types/src/codec/optimization.rs— open contribution and summary wire contracts.crates/core/src/api/optimization.rs— bounded recorder and close-time arithmetic.crates/core/src/api/llm.rsandcrates/core/src/stream.rs— buffered/streaming lifecycle integration.crates/core/src/observability/{atif,otel,openinference}.rs— exporter projections.crates/core/tests/integration/middleware_tests.rs— ordering, mark parentage, and end-summary integration.Review guidelines
Testing summary
cargo test --workspaceafter rebasing onto currentmain: passed, including 825 core library, 75 FFI unit, 65 FFI integration, 63 PII redaction, 57 middleware, 26 worker integration, 15 stream integration tests, and all doctests.cargo clippy --workspace --all-targets -- -D warnings,cargo deny check, formatting, Ruff,ty, Go vet, Node formatting/docstrings, lockfile, protobuf, FFI-header, and non-attribution pre-commit checks passed.Compatibility and scope
optimization_contributionsremain wire-compatible._v2helper accepts contribution JSON.Review feedback addressed
src/following repository convention.LlmOptimizationKindopen and string-backed so third-party kinds do not require a Relay release; standard constants retain exact wire values.appliedfield fail-safe and consistent across Rust, Go, and Python; only explicit applied evidence can affect accounting.extradata from reappearing on the wire.Related PRs
Summary by CodeRabbit
New Features
optimization_summaryfor buffered and streaming calls.optimizationContributionsacross Node, Python, Go, and native/FFI; addedoptimization_summary/optimization_contributionsaccessors and OpenInference/OTEL/ATIF optimization attribute exports.Bug Fixes
Tests