feat: add Rhesis integration for OpenTelemetry tracing - #3669
Open
Arman-Beykmohammadi wants to merge 40 commits into
Open
feat: add Rhesis integration for OpenTelemetry tracing#3669Arman-Beykmohammadi wants to merge 40 commits into
Arman-Beykmohammadi wants to merge 40 commits into
Conversation
Add the rhesis-haystack integration, providing a RhesisConnector component and an OpenTelemetry-based tracer that exports Haystack pipeline, component, and agent spans to Rhesis. Includes the Haystack-to-Rhesis semantic mapping layer, packaging (pyproject, pydoc config, py.typed), examples, and tests. Co-authored-by: Cursor <cursoragent@cursor.com>
…ntry Wire the new Rhesis integration into repo automation: add the test workflow, labeler rule, coverage-comment trigger, and the README integrations table row. Co-authored-by: Cursor <cursoragent@cursor.com>
Arman-Beykmohammadi
requested review from
davidsbatista
and removed request for
a team
July 27, 2026 05:47
|
|
github-actions
Bot
added
topic:CI
type:documentation
Contributor
|
@Arman-Beykmohammadi do you want to keep this PR open or move into your own repo? |
Author
|
@davidsbatista |
Haystack 3.0 removed `serialize_class_instance` and
`deserialize_class_instance` from `haystack.utils.base_serialization`, so
importing them made `RhesisConnector` fail to import on 3.0.
The span handler is an arbitrary user class rather than a component, so it
still needs the type-tagged `{"type": ..., "data": ...}` envelope those
helpers produced. Rebuild it locally on top of
`generate_qualified_class_name` and `import_class_by_name`, keeping the
serialized format byte-compatible so pipelines serialized before this
change keep deserializing.
Deserialization now also rejects a type that is not a `SpanHandler`
subclass, instead of failing later on a missing `from_dict`.
Haystack 3.0 moved the Agent loop off `Pipeline._run_component`: every iteration now opens its own `haystack.agent.step` span, and the LLM call and each tool call are traced directly as `haystack.agent.step.llm` and `haystack.agent.step.tool` instead of going through a `ToolInvoker` component span. Those spans carry none of the `haystack.component.*` tags, so the existing span-kind rules — which all key off component type — left them unresolved. An agent's LLM and tool work was exported as untyped `function.*` spans with no model, token, or tool attributes attached. Add an `_OPERATION_ONLY` rule form that matches on operation name alone, and map the new operations to `ai.llm.invoke`, `ai.tool.invoke`, and a `function.haystack.agent.step` grouping span. `RhesisSpan` now retains its operation name, which is the only discriminator available at enrichment time and is what lets reply metadata be promoted from the in-agent LLM span; that promotion moves into `_apply_chat_reply_metadata`, shared with the existing `ChatGenerator` component path. Tool arguments and results are stamped as `ai.tool.input`/`ai.tool.output` content rather than `ai.prompt`/`ai.completion` events, since they are not a prompt. The haystack 2.x `ToolInvoker` rules are kept, so 2.x pipelines trace exactly as before.
Haystack models delegation to a specialist agent as an ordinary tool call: the tool's function runs `Agent.run`, so the nested agent shows up as an `ai.agent.invoke` span under a plain `ai.tool.invoke` parent. Nothing in the exported trace said the call was a handoff, or which agent handed off to which. When an agent span opens directly inside a tool span, re-label that still-open tool span as `ai.agent.handoff` and stamp `ai.agent.handoff.to` with the tool name and `ai.agent.handoff.from` with the enclosing agent — the nearest ancestor tool span for a specialist, or the nearest component name for a top-level agent. The nested agent span itself picks up `ai.agent.name`. Add an end-to-end test that runs a coordinator agent delegating to a specialist through a tool, driven by a scripted chat generator, and asserts the shape of the exported span tree: a single pipeline root, one `agent.invoke` per agent, an `llm.invoke` per step, and the nested agent parented by the handoff span.
A pipeline root span stamped `conversation.input`/`conversation.output` by
running the entire pipeline input and output mappings through
`_stringify_content`. A turn was therefore displayed as a serialized dict —
`{"chat": {"messages": [...]}}` — rather than what the user asked and what
the pipeline replied.
Walk the per-component payloads instead and take the last user message and
the last assistant message from whichever component carries a chat history,
preferring an Agent's `last_message` when it reports one. Assistant turns
that only request tool calls carry no text, so they are skipped rather than
ending the search.
When no chat messages can be found the span now stamps no conversation text
at all. A serialized payload is never a valid rendering of a turn, so showing
nothing is the better failure mode.
Note that this is a fallback for pipelines traced with no Rhesis SDK endpoint
above them, not an authoritative record: only the application knows how it
derives its reply.
The role-matching logic is factored out and reused by the agent-span
extraction, which as a result also skips text-less messages instead of
stopping at the first one.
… owns it When a Haystack pipeline runs inside a Rhesis SDK `@endpoint` / `@observe` call, both the SDK root span and the Haystack root span stamped `is_turn_root`. Only one span per turn may carry that flag, so the exporter stripped the Haystack root's real parent and the whole Haystack subtree detached into a second conversation turn — with the pipeline payload restated as that turn's input and output. Decide turn ownership once, at span creation, from the SDK's `get_root_trace_id` context var: when the SDK has already opened the turn root, the Haystack root span is a child of that turn rather than a turn of its own. Such a span drops the `is_turn_root` flag and skips conversation input/output stamping altogether, since the SDK span already carries the mapped user message and reply. Session and conversation ids are kept — they are useful on a nested span and the exporter propagates them anyway. The check is deliberately keyed on the SDK context rather than on `trace.get_current_span()`, so that unrelated ambient instrumentation (an HTTP server span, say) cannot leave a turn with no root at all. Standalone Haystack, with nothing above it, still owns the turn.
Every published `rhesis-sdk` version from 0.9.1 on declares `requires-python >= 3.12`, so the Python 3.10 CI jobs could not even resolve the environment. Raise `requires-python` to match the SDK and move the test matrix (plus the lint and coverage jobs pinned to the lowest supported version) to 3.12. Raising the floor also raises ruff's inferred target version, which enables UP042 on `MappingPromotion`; `StrEnum` is available from 3.11. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tree this test asserts — a span per agent step, a span per tool call, and the handoff promoted from the tool span that runs a nested Agent — only exists from Haystack 3.0 on. On 2.x an agent routes all of a step's tool calls through a single batched ToolInvoker component span, so the test failed in the "lowest direct dependencies" job, which resolves haystack-ai to the declared floor of 2.22.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lint job installs the latest ruff into a fresh environment, so it picked up two ruff 0.16 changes the moment the job started running: - `PLR0917` (too many positional arguments) left preview and now flags `RhesisConnector.__init__`. Ignore it, as every other integration in the repo that selects `PLR` already does, next to the `PLR0913` argument-count rule it complements. - `ruff format` now formats Python code blocks inside Markdown, which wants blank lines around the README's top-level definitions. Neither surfaced before: on Python 3.10 the lint job failed while resolving the environment, before ruff ever ran. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`RhesisConnector` covers the common case: add it to a pipeline and every run is traced. An application that owns its own loop — a chat REPL, a batch script, a server handling one turn per request — needs two things a component sitting inside the pipeline cannot provide: tracing switched on without a pipeline to attach to, and a span wrapping a whole pipeline run so that a conversation turn has a root of its own. Without that root span the Haystack pipeline span claims the turn and reports the serialized pipeline input and output as the conversation text, so the Rhesis conversation view shows a dict dump instead of what the user asked and what the application replied. Add `RhesisTracing`, which constructs the connector for its side effects and never adds it to a pipeline. `turn()` opens the turn root, stamps the conversation input, and sets the SDK's `root_trace_id` context var so the Haystack tracer defers instead of claiming the turn. Turns after the first attach to a synthetic non-recording parent carrying the first turn's trace id — the approach the Rhesis SDK uses for turns it serves itself — so a conversation reads as one trace rather than one per exchange. The exporter strips that placeholder parent, so every turn is still stored as a root span. The reply is assigned by the caller through `ConversationTurn.output`: only the application knows which part of a pipeline result is the user-facing answer, which may be a tool result or a value held in agent state rather than the last assistant message. Construction never raises. A missing `RHESIS_API_KEY`, a rejected configuration, or an explicit `enabled=False` all yield an inert instance whose `turn()` still works, so an application runs untraced rather than failing to start and callers need no branching.
`HAYSTACK_CONTENT_TRACING_ENABLED` is read once, when `haystack.tracing.tracer` is first imported, so setting it at test-module scope only had an effect when that module happened to be collected before anything else imported Haystack. It never did: `test_agent_span_tree.py` imports Haystack first, so both `os.environ[...]` lines were dead and the suite ran with content tracing off while appearing to run with it on. Patch the resolved flag from an autouse fixture instead, which is order-independent, and drop the two dead assignments. Tests that need the flag off already patch it back themselves. Also uninstall the global tracer after every test: `RhesisConnector.__init__` calls `tracing.enable_tracing` as its last statement, so a test that builds one left it installed for the rest of the session. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
`RhesisConnector` installs one tracer process-wide via `tracing.enable_tracing`, but the tracer stored the current trace id as instance state and wrote it on every root span. Under concurrent pipeline runs — hayhooks, FastAPI, AsyncPipeline — the last writer won, so `RhesisConnector.run()` could return another request's trace id and hand the user a deep link into someone else's trace. Publish it on a ContextVar for the lifetime of the root span instead, which is exactly the window in which the connector component runs and reads it back. `test_concurrent_span_stacks` cannot catch this: it drives two separate tracer instances, so the shared-state case never arises. The new test drives two concurrent root spans through one tracer and asserts each sees its own id. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
Two defects in the connector's `invocation_context` socket.
The leak: `run()` sets a ContextVar from inside the pipeline and a component has
no teardown hook, so the value was still set when the next run's root span read
it. Measured over four consecutive runs of one pipeline, passing alice, nothing,
bob, nothing:
alice -> alice bob -> bob
<none> -> alice <none> -> bob
In a server that is one user's turn filed under another user's conversation.
`RhesisTracer.trace` now takes a restore point around each root span, so a run
sees what was scoped outside the pipeline plus whatever it supplied itself, and
nothing survives the root span's close.
The silent drop: the mapped attributes were only stamped alongside extracted
conversation text, so a pipeline carrying no chat messages discarded
`invocation_context` altogether — no session id, no test-run correlation. Apply
them on the root span whether or not conversation text was found, which is also
what the create_span path already did.
Both applications now go through one helper, so the socket path and
`rhesis_invocation_context()` produce identical attributes.
Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
`HAYSTACK_RHESIS_ENFORCE_FLUSH` defaults to true and the flush ran in every span's `finally`, calling `provider.force_flush(timeout_millis=30_000)`. A ten-component pipeline therefore blocked on ten synchronous round trips, and the SDK's BatchSpanProcessor — configured with `max_export_batch_size=512` — never had more than one span to batch. Gate it on the root span. The default keeps its guarantee that everything is on the backend by the time the run returns, at one export instead of ten. Langfuse inherits the same env var and default, but its `flush()` hands work to a client-side queue; forcing an OTel provider flush costs materially more, so matching the flag name is right and matching the per-span frequency is not. The README presented per-component flushing as the safe choice. Correct it: name the cost, note that OpenTelemetry's atexit hook already covers normal exit, and list the three cases where the default genuinely earns its keep. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
…t in `base_url`, `environment`, `project_id` and `frontend_url` were resolved from the environment in `__init__` and then written into `to_dict`. A pipeline dumped on a laptop with `RHESIS_BASE_URL=http://localhost:8080` carried that URL into the YAML, so deploying the definition pointed production at localhost with no way for the target environment to correct it. Serialize the arguments as passed and leave anything the caller delegated to the environment as `None`, so `from_dict` resolves it wherever the pipeline actually runs. This is what `Secret.from_env_var` already does for the API key: serialize the reference, not the resolved value. The previous test asserted the resolved defaults, which locked the behaviour in. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
… itself `test_mapping.py` asserted that `HAYSTACK_TAG_MAPPING` contained the keys in `APPENDIX_A_HAYSTACK_TAGS` — but both tables were written from the same list, so the assertion held no matter what the tracer emitted. It could only fail if someone edited one table and forgot the other. The tables themselves had no runtime consumer: `HAYSTACK_TAG_MAPPING`, `HAYSTACK_OPERATION_MAPPING`, `MappingTarget`, `MappingPromotion` and the two APPENDIX_A frozensets were read by that test alone, ~190 lines of them. "Appendix A" also names an internal design document a deepset maintainer cannot resolve. Deleted; README.md keeps the readable version of the mapping. Replaced by `test_emitted_spans.py`, which runs real pipelines through an InMemorySpanExporter and asserts the span names, promoted attributes and events that actually leave the process. That test caught two defects on its first run; the first is fixed here. - Span names were `AIOperationType` members rather than their values. `AIOperationType` is a `(str, Enum)`, not a `StrEnum`, so a span name rendered as "AIOperationType.LLM_INVOKE" anywhere it was formatted into text. Equality, dict lookups and the OTLP encoder all resolve to the value, which is why it went unnoticed. The tables now hold `.value`, as does the `update_name` call that relabels a tool span as an agent handoff. Also drops three `SPAN_KIND_RULES` rows that existed only to be skipped — root naming is a lookup of its own now — and the `component_type == rule_type` branch after an `endswith` that already matched it. Removing `MappingPromotion` removes the package's only `enum.StrEnum`, which requires Python 3.11 and blocks the return to the repo's 3.10 floor. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
…s answer `_messages_from_agent_payload` only looked for a `messages` key, but a ChatGenerator publishes its answer on `replies`. A prompt-builder-into-generator pipeline — the shape of the README quickstart and of both shipped examples — has no `messages` anywhere in its output, so `rhesis.conversation.input` carried the user's question and `rhesis.conversation.output` was never set at all. Every conversation from a plain RAG or chat pipeline was half a turn. Found by the emitted-span test added in the previous commit, which is the case the deleted table-completeness test could not reach. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
Three changes to how the package is laid out, no behaviour change. `_haystack_tags.py` now owns the Haystack span-tag and operation literals. Six of them — the component name and type, the agent run, the two agent-loop steps and the tool name — were declared independently in both `tracer.py` and `mapping.py`. Two sources of truth for the same strings, in a package whose whole job is getting those strings right. `_extraction.py` takes the seven helpers that pull conversation text out of Haystack payloads. They are one job, and none of it is Haystack-to-OTel bridging, which is what `tracer.py` is for. `DefaultSpanHandler.handle` did five unrelated things in one 67-line method; it is now four named steps — `_apply_invocation_context`, `_promote_conversation_io`, `_rename_tool_invoker`, `_apply_model_metadata` — each carrying the reasoning that used to be an inline comment block. Also folds in three smaller cleanups from the same review: - `MAX_CONTENT_LENGTH` is imported from the SDK, which already declares it as framework-agnostic, instead of being redefined here. The two bounds that sit next to each other now say why they differ. - `RhesisTelemetry` resolves `frontend_url` in `__post_init__`, so `get_trace_url` stops re-resolving a value the connector had already resolved, and a directly constructed telemetry gets the same treatment. - `RhesisSpan.close()` replaces `RhesisTracer._close_span` reaching across the class boundary for `span._context_manager`. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
… differ `B008` was ignored package-wide for a single `Secret.from_env_var` default; langfuse puts an inline `# noqa: B008` on its two, which is narrower. `B017` existed to permit one `pytest.raises(Exception)`. Naming the exception turned up that the connector's own "RHESIS_API_KEY is required" error is unreachable when the caller passes a strict `Secret.from_env_var`: Haystack rejects the unset variable first. The two paths now have a test each and assert their real messages. Also writes down the asymmetry a reviewer will otherwise ask about: `RhesisConnector` raises on a missing key while `RhesisTracing` degrades to a no-op. That is deliberate — the connector is a component the user wired into a pipeline, so failing loudly is the honest signal; `RhesisTracing` wraps an application's own loop and is not in its data path. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
Three gaps, all in the same place. `conversation.py` was missing from the pydoc loader, so `RhesisTracing` and `ConversationTurn` were exported public API with no published reference, and the README's Conversations section pointed at nothing. `rhesis_invocation_context` was exported and mentioned nowhere. It is the only way to attach session or test metadata to work that is not a pipeline run, so without it a standalone Agent has no correlation at all — and no user was going to discover it from the export list. There was no agent example, which is the case the integration handles best: one constructor call, no pipeline, and a complete `ai.agent.invoke` tree with a span per step and per tool call. `example/agent.py` shows it with two tools. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
The mapped attributes were written only inside the `if context.is_root:` branch, so a user filtering a trace's child spans by their own run id or session found nothing — which is most of the reason to pass one. Langfuse gets the same effect from `propagate_attributes`. Stamp them on every span. The turn-root flag stays on the root alone: on a child the exporter strips its real parent and the subtree detaches into a turn of its own. Coverage differs by how the context was supplied, and the README now says so. `rhesis_invocation_context` is set before the run starts, so no span opens without it. The connector's input socket supplies its value from inside the run, so a component whose span closed before the connector executed was already exported — the root span always gets it, since it closes last, which is what conversation grouping needs. `user_id` and `tags` need no special handling: unmapped keys already travel as `haystack.invocation.<key>`, on every span now. Promoting them to first-class `ai.*` names would mean inventing wire names the Rhesis backend does not index, so that stays an SDK decision. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
…ge history On haystack 2.x an agent's tool calls arrive batched through one `ToolInvoker` component span, whose input is the entire message history and whose output the entire tool-message list. The span was renamed after the tools it called but its content was left as the serialized dump — unreadable in a trace viewer and impossible to index by tool name. Replace the content with the calls (`id`, `name`, `arguments`) and their results (`id`, `name`, `arguments`, `result`, `error`), on the existing `ai.tool.input.content` / `ai.tool.output.content` attributes. langfuse does the same, for the same reason. Gated on the content flag, since it is content; the rename still happens either way, because that is structure. haystack 3.0 needs none of it — the agent loop already opens an `ai.tool.invoke` span per call — but `haystack-ai>=2.22.0` is the declared floor, so 2.x users are in scope. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
…ad none `test_agent_span_tree.py` was 226 lines holding one test, over the most intricate logic in the package — per-tool spans, handoff promotion, agent nesting. When it failed it did not say which part broke. The run now happens once in a fixture and each claim is its own test, so a failure names itself. Three gaps closed: - A standalone `Agent`, traced with no pipeline around it. The connector is built and never used again, so `ai.agent.invoke` is the trace root rather than a child of the pipeline span. This is the case the integration handles best and had no coverage at all. - The async path. `haystack.async_pipeline.run` was exercised only as a string in a mapping table, never end to end. Covered through whichever entry point the installed Haystack offers — 3.0 folded `AsyncPipeline` into `Pipeline.run_async`, 2.x has the separate class. - Root-span enrichment on the async path, which nothing checked. Also drops the local `traced_exporter` fixture now that conftest provides one. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
Both were introduced by earlier commits on this branch and only show up in the "unit tests with lowest direct dependencies" job, which resolves haystack-ai 2.22.0 — so the 3.0 matrix cells stayed green and hid them. `MAX_CONTENT_LENGTH` was imported from `rhesis.sdk.telemetry.attributes` when the local redefinition was dropped, but that constant does not exist in rhesis-sdk 0.9.1, the declared floor. The import failed at collection, taking the whole suite with it. Raise the floor to >=0.12.0, which is what CI actually resolved and what was tested all along; 0.9.1 also pins mcp==1.26.0 and ships the diskcache dependency whose unsafe-pickle CVE is called out in this package. The async tests built a `Pipeline` and then moved its components into an `AsyncPipeline` on 2.x, which Haystack refuses — "Components can't be shared between Pipelines". Choose the right class up front instead. Verified on all four CI configurations: py3.14 + haystack 3.0.0 106 passed py3.12 + haystack 3.0.0 106 passed py3.12 + haystack 2.22.0 95 passed, 11 skipped (lowest-direct) py3.12 + haystack main 106 passed Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
Contributor
|
Hi @Arman-Beykmohammadi, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
HaystackBot
added
the
cla-pending
HaystackBot
marked this pull request as draft
August 12, 2026 16:36
6 tasks
HaystackBot
marked this pull request as ready for review
August 12, 2026 17:09
HaystackBot
removed
the
cla-pending
Contributor
|
Thanks for signing the CLA, @Arman-Beykmohammadi! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
Open
2 tasks
rhesis-sdk brings an evaluation stack a tracer has no use for. Measured with uv pip compile for linux-x86_64: haystack-ai alone resolves to 43 packages, with rhesis-sdk 201 — torch, 15 nvidia-* CUDA packages, a hard-pinned deepeval==3.7.0 that would conflict with deepeval-haystack, and the LangChain/LangGraph stack. All of it for six symbols that depend on nothing heavier than stdlib and rhesis.telemetry.schemas. It also forced requires-python = ">=3.12" on a repo where 93 of 99 integrations require >=3.10. Those six symbols now live in the lightweight rhesis package, so depend on rhesis[telemetry] and import them from their canonical rhesis.telemetry.* paths. That resolves to 52 packages, against 54 for langfuse. It also makes rhesis.telemetry a declared direct dependency rather than something reached through rhesis-sdk. The Python floor drops to >=3.10 with it and the CI matrix returns to the ["3.10", "3.14"] its peers use. While the matrix was open: the integration step is guarded to a single cell the way langfuse guards its own, and RHESIS_API_KEY moves from workflow-level env into that step, so no other step can read it. Verified on 3.10 and 3.14, and against the declared floor of haystack-ai 2.22.0 on 3.10. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
…global get_tracer_provider caches one provider per process and installs it as the OpenTelemetry global. Both are wrong for a library. The global means whichever of Rhesis and the host application's own instrumentation initialises first wins — OpenTelemetry refuses the loser's override — so adding this component to a pipeline could silently redirect a user's Datadog spans to Rhesis, or lose Rhesis spans to Datadog. The cache means a second connector with a different project_id quietly exports to the first one's project, because the exporter that stamps project_id is built with the first provider. build_tracer_provider, added upstream for embedded callers, has neither property. Nothing here needed the global: spans are opened through telemetry.otel_tracer and flushed through telemetry.provider, and parent-child nesting travels in the OpenTelemetry context, which is shared across providers — so a pipeline running inside a Rhesis SDK @endpoint span still nests under it. One caller did depend on the global, quietly: RhesisTracing opened its turn spans with trace.get_tracer(). Left alone it would have kept working and recorded nothing, since the global tracer is a no-op unless someone claims it — every turn root vanishing while its children still exported. It now borrows the connector's tracer through a public RhesisTracer.telemetry property, and RhesisTelemetry is exported, since it is also the declared type of SpanHandler.tracer and custom handlers should not have to import a private module path for it. Verified end to end: a host that sets up its own provider first keeps the global, its own span reaches only its exporter, and the Haystack spans reach only Rhesis. Three RhesisTracing turns in the same process export under one trace id with the pipeline spans nested inside them. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
run() writes invocation_context to a ContextVar that RhesisTracer.trace scopes: it sets a restore point when the run's root span opens and restores it when that span closes, so a write from inside the run cannot outlive it. That covers the pipeline path, and a regression test pins it. It does not cover run() called with no root span open — a connector invoked directly rather than as part of a pipeline run. There is no restore point there, so the value stayed set for the rest of the process and became the default for every later run that supplied none: one caller's session id attached to another caller's conversation. Honour the context only while a root span is open, and say so when it is dropped, pointing at rhesis_invocation_context, which scopes the value to its own block and is the right tool outside a pipeline run. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
…year rhesis.md is pydoc output. No other integration commits one: the docusaurus sync workflow runs `hatch run docs` at release time and pushes the result into the haystack repo, so a copy here is a stale duplicate of a build artifact from the moment it lands. The SPDX headers said 2023-present, copied from langfuse, on files written in 2026. Matches what the most recent new integration used for its own files. Signed-off-by: Arman Beykmohammadi <arman.beykmohammadi@rhesis.ai>
This was referenced Aug 14, 2026
Open
Arman-Beykmohammadi
force-pushed
the
feat/rhesis-integration
branch
from
August 14, 2026 11:36
3d6d598 to
e8cdf43
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
Adds
rhesis-haystack, a tracing integration that exports Haystack pipeline, component andAgentspans to Rhesis over OpenTelemetry.
What Rhesis is. An open-source platform for structured feedback and evaluation on LLM agents.
Domain experts review agent responses in a UI; the feedback stays attached to the test case and the
agent version that produced it, and recurring feedback becomes tests and metrics that run on every
change. Docs: https://docs.rhesis.ai
How this relates to the tracers already here. The repo has four.
datadog-haystackandopentelemetry-haystackexport to generic OTel/APM backends;langfuse-haystackandweave-haystackare LLM trace viewers. This one is closest in shape to
langfuse-haystack— I followed its filelayout and class split deliberately (
Connector/Tracer/Span/SpanContext/SpanHandler/DefaultSpanHandler), so it should read the same to anyone who maintains that one.What it adds beyond viewing traces is correlation. Spans carry:
rhesis.test.run_id,rhesis.test.id,rhesis.test.result_idrhesis.conversation.id,rhesis.conversation.is_turn_rootSo the use case is running a Haystack pipeline under a Rhesis test run and having reviewer feedback
land on the exact span tree that produced the answer — rather than one more place to look at traces.
Dependency weight. Resolved with
uv pip compilefor linux-x86_64:haystack-aihaystack-aialone (py3.10)haystack-ai+rhesis[telemetry]>=0.13.0(this PR, py3.10)rhesis, sixopentelemetry-*,protobuf,googleapis-common-protoshaystack-ai+langfuse>=4.0.0(for reference, py3.10)The lightest of the four tracers in the repo, and
requires-python = ">=3.10", matching 93 of the 99integrations on main. (For contrast, depending on the full
rhesis-sdkinstead resolves to 201packages and forces
>=3.12: it bringstorch, 15nvidia-*CUDA packages, a hard-pinneddeepeval==3.7.0that would conflict withdeepeval-haystack, and the LangChain/LangGraph stack.None of it is used by a tracer, so the six symbols this integration needs were moved into the
lightweight
rhesispackage upstream.)No process-wide side effects.
RhesisConnectorbuilds its ownTracerProviderand never callstrace.set_tracer_provider, so a user who already runs Datadog or their own OTel pipeline keeps theglobal provider and every span it produces. Nesting still works across the two, because parent-child
relationships travel in the OpenTelemetry context rather than in the provider.
Contents:
RhesisConnector— add to a pipeline with no wiring to other components; returnsname,trace_url,trace_idRhesisTracer/DefaultSpanHandler— bridges Haystack spans to OTel, with the sameSpanHandlerextension point as
langfuse-haystackai.operation.type, content and token attributes,invocation-context propagation
ToolInvokercomponent span and the 3.0 agentloop (
haystack.agent.step.*), including promoting a tool span to an agent handoff when a tool runsan
AgentRhesisTracing— conversation-aware entry point for apps that drive Haystack outside a pipeline(chat servers, REPLs)
pyproject.toml, pydoc config,py.typed, examples, README, unit andintegration tests
.github/workflows/rhesis.yml, labeler rule, coverage-comment trigger, READMEintegrations table row
How did you test it?
hatch run fmt-check— passeshatch run test:types— passes (mypy, 8 source files)hatch run test:unit— 109 passed, on Python 3.12 and on Python 3.10 (the new floor)haystack-ai==2.22.0on Python 3.10 — 98 passed, 11 skipped (theagent-loop span-tree tests require Haystack 3.0 and skip below it)
hatch run test:integration— passes against a local Rhesis backend. NeedsRHESIS_API_KEY, so itskips in CI until the secret is added.
TracerProvideras the OTelglobal first, then constructs
RhesisConnectorand runs a pipeline. The host keeps the global,its
http.requestspan goes only to its own exporter, and the Haystack spans go only to Rhesis.Multi-turn
RhesisTracingin the same process: three turn spans exported under one trace id, withthe pipeline spans nested inside them and
is_turn_rooton the turn roots only.Agentwith two tools and a nested agent-as-tool.Notes for the reviewer
On the red checks. All of them fail for one reason, and it isn't the code: this PR depends on
rhesis[telemetry]>=0.13.0, which is not on PyPI yet. The symbols the integration needs were movedinto that lightweight package upstream, and its release is a few days out. Nothing gets as far as
running a test — the environment can't be resolved, and the license job reports
rhesis:0.13.0 → Errorfor the same reason. I've verified the post-release state against a locally built 0.13.0wheel:
--resolution lowest-directresolves tohaystack-ai==2.22.0+rhesis==0.13.0, 109 unittests pass on Python 3.10, and mypy is clean. I'll re-run CI the moment the release lands, and I'd
suggest not spending review time on the failures until then.
Then three things I'd like your read on:
tracer.pyandmapping.pyhold the Haystack-to-OTel bridge.DefaultSpanHandler.create_spanand.handlecontain all the version branching and are wherereview is most valuable.
_extraction.pyand_haystack_tags.pyare small support modules.RhesisTracinginconversation.pyenables tracing without a pipeline,for apps that drive Haystack from their own loop. Without a turn root, the pipeline span claims the
turn and reports serialized dicts as the conversation text. No other tracer ships a non-component
entrypoint, so tell me if you'd rather it lived outside this repo — the connector works without it.
ai.tool.invokespans and the agent-handoff promotion are tested end-to-end intest_agent_span_tree.py.Checklist