Multitenancy support: basic CPU/Memory limits, tenant identifiers, limiter extensions

[jmacd]

Change Summary

Replaces #3389, incorporating the feedback.

This is better aligned with dataflow engine Policies framework.

Major renaming:

• tenant_token instead of tenant_descriptor
• extractors instead of actions

The existing document was split into two parts, the -tenants.md and -limiters.md documents.

A new document on memory management policies is added, and a new overview links to the other three.

What issue does this PR close?

• Closes Design and implement logical retained-work memory budgeting #3272

• Closes Pipeline physical resource limits #1245

• This is a documentation-only PR.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Status last refreshed: 2026-07-25 22:03:13 UTC.

• Waiting on: Author
• Next step: Investigate required status check failures.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected. If you believe this pull request is incorrectly routed as waiting on the author, comment /dashboard route:reviewers to route it from waiting on the author to waiting on reviewers. If the last refreshed time above predates your latest reply or push, the dashboard hasn't processed it yet.

[Copilot]

Pull request overview

Adds a set of design documents for multitenancy in the OTAP dataflow engine, covering tenant identification (“tenant tokens”), limiter policy configuration, and memory-management concepts, plus supporting diagrams.

Changes:

• Introduces a multitenancy overview doc and three focused design docs: tenant identity, limiter policies, and memory management.
• Adds two SVG diagrams to illustrate tenant-token request flow and regulated memory accounting.
• Documents example YAML configuration patterns for CPU/memory policies and conditional tenant-based limiter behavior.

Reviewed changes

Copilot reviewed 4 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
rust/otap-dataflow/docs/multitenancy-tenant.md	New design doc for tenant token extraction, propagation, and condition lookup model.
rust/otap-dataflow/docs/multitenancy-tenant-diagram.svg	Diagram for request-context/token resolution and limiter flow.
rust/otap-dataflow/docs/multitenancy-overview.md	New overview doc linking to the detailed multitenancy design docs; includes CPU policy examples.
rust/otap-dataflow/docs/multitenancy-memory.md	New design doc for memory regulation concepts and example configuration.
rust/otap-dataflow/docs/multitenancy-memory-diagram.svg	Diagram illustrating regulated vs unaccounted memory and configured limits.
rust/otap-dataflow/docs/multitenancy-limiters.md	New design doc for rate/resource limiter policy schema, tenancy conditions, and operational behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.75%. Comparing base (474998a) to head (1a04763).
⚠️ Report is 97 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3583      +/-   ##
==========================================
+ Coverage   86.23%   86.75%   +0.51%     
==========================================
  Files         739      784      +45     
  Lines      292355   319867   +27512     
==========================================
+ Hits       252116   277502   +25386     
- Misses      39715    41841    +2126     
  Partials      524      524              

Components	Coverage Δ
otap-dataflow	87.79% <ø> (+0.50%)	⬆️
query_engine	89.57% <ø> (-0.01%)	⬇️
otel-arrow-go	52.45% <ø> (ø)
quiver	92.20% <ø> (+0.01%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.