feat: Add pressure-aware receiver rate throttling

[lalitb]

Change Summary

Implements RFC 0002 pressure-aware receiver rate throttling.

This adds a receiver-local, lock-free rate limiter driven by the existing memory pressure signal. V1 supports:

• OTLP request_bytes/second, measured as decompressed OTLP payload bytes
• Syslog / CEF messages/second, measured per UDP datagram or emitted TCP/TLS record fragment
• receiver_instance aggregation
• activation at soft pressure and above
• enforce and observe_only modes

OTLP returns protocol-native retryable responses when throttled: HTTP 503 with Retry-After, and gRPC RESOURCE_EXHAUSTED with grpc-retry-pushback-ms. Syslog / CEF drops over-limit messages/fragments while keeping TCP/TLS connections open; hard memory pressure behavior is unchanged.

The PR also adds receiver capability validation through ReceiverFactory::supported_rate_units, so unsupported receiver/unit combinations fail closed at startup and at engine receiver construction time.

V1 limitations called out in docs:

• no tenant fairness yet; rate state is per receiver instance
• no per-receiver opt-out inside a scoped policy
• mixed-receiver pipelines must scope rate_limit to receivers that support the configured unit
• journald, ETW, user_events, host_metrics, topic, internal telemetry, and traffic generator do not opt into rate throttling in this PR

What issue does this PR close?

• Closes [chore] docs: add pressure-aware EPS throttling RFC #3484

How are these changes tested?

Unit and integration coverage added for:

• token-bucket/GCRA admission, bounded debt, burst handling, recovery, observe-only behavior, and high-rate rounding edge cases
• OTLP HTTP rate refusal, including early fast-fail before semaphore/body collection
• OTLP gRPC rate refusal, including RESOURCE_EXHAUSTED and retry pushback metadata
• gRPC rate fast-fail before concurrency layers
• syslog UDP rate refusal
• syslog TCP/TLS rate refusal and oversized-fragment accounting
• rejected oversized syslog fragments discarding through newline
• receiver capability validation for supported and unsupported units
• config validation for missing memory pressure source, invalid intervals, byte-unit parsing, and live memory-limiter mutation rejection

Local validation run:

• cargo fmt --all
• cargo check -p otap-df-otap
• cargo check -p otap-df-core-nodes
• cargo check -p otap-df-controller
• targeted cargo test runs for rate limiter, OTLP, syslog, and live-control cases
• npx markdownlint-cli2 on changed Markdown files
• python3 tools/sanitycheck.py

Benchmark details:

Hardware:

• MacBook Pro, Apple M4 Pro
• 12 cores: 8 performance, 4 efficiency
• 24 GB memory
• macOS 26.5.2, arm64

Happy-path local throughput comparison:

• main, rate limiting absent: 198,878 logs/sec, stdev 170
• PR branch, rate limiting not configured: 198,922 logs/sec, stdev 93
• PR branch, rate limiting configured but no pressure: 198,868 logs/sec, stdev 11

Result: no measurable happy-path throughput regression in this local benchmark.

Are there any user-facing changes?

Yes.

This adds new pressure-aware rate throttling policy support for OTLP and Syslog / CEF receivers, new receiver refusal metrics, and new documented syslog rate-drop behavior.

This is also a source-breaking embedding API change: custom receiver embedders must declare supported rate-limit units on ReceiverFactory.

Changelog

• Added a .chloggen/*.yaml entry
• This PR is a chore (indicated in title)
• This is a documentation-only PR.

[Copilot]

Pull request overview

This PR implements RFC 0002 pressure-aware receiver rate throttling for otap-dataflow, introducing a receiver-local (lock-free) rate limiter driven by the existing process memory pressure signal. It also adds capability-based validation so unsupported receiver/unit combinations fail closed during startup and engine receiver construction.

Changes:

• Add a receiver-local token-bucket/GCRA-style rate limiter and a gRPC fast-fail Tower layer, plus OTLP/Syslog receiver integrations (including retryable OTLP responses and Syslog drop behavior).
• Extend config/policy resolution and validation to support a rate_limit policy, including byte-unit parsing for request_bytes/second and semantic validation requiring a configured memory limiter.
• Add receiver capability metadata (ReceiverFactory::supported_rate_units) and enforce it in controller startup validation and engine receiver construction.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
rust/otap-dataflow/src/main.rs	Updates controller startup validation call sites for the new rate-limit policy parameter.
rust/otap-dataflow/docs/memory-limiter-phase1.md	Documents soft-pressure interaction with receiver admission policies and adds new rate-limit metrics/events.
rust/otap-dataflow/docs/admin/live-reconfiguration.md	Documents that runtime memory limiter mutation is rejected.
rust/otap-dataflow/crates/otap/tests/pipeline_tests.rs	Updates pipeline construction tests for new pipeline build parameter.
rust/otap-dataflow/crates/otap/tests/internal_telemetry_metrics.rs	Updates internal telemetry tests for new pipeline build parameter.
rust/otap-dataflow/crates/otap/tests/durable_buffer_processor_tests.rs	Updates durable buffer tests for new pipeline build parameter.
rust/otap-dataflow/crates/otap/tests/core_node_liveness_tests.rs	Updates liveness tests for new pipeline build parameter.
rust/otap-dataflow/crates/otap/src/rate_limiter.rs	Adds the receiver-local pressure-aware rate limiter implementation + unit tests.
rust/otap-dataflow/crates/otap/src/rate_limit_layer.rs	Adds a Tower layer for gRPC fast-fail when the rate bucket is exhausted.
rust/otap-dataflow/crates/otap/src/otlp_metrics.rs	Adds OTLP receiver counters for refused/would-refuse rate limiting.
rust/otap-dataflow/crates/otap/src/otlp_http.rs	Adds OTLP HTTP rate-limit refusal paths (503 + Retry-After) and accounting/metrics updates.
rust/otap-dataflow/crates/otap/src/otap_grpc/otlp/server_new.rs	Adds OTLP gRPC rate-limit refusal handling (RESOURCE_EXHAUSTED + retry pushback) and metrics updates.
rust/otap-dataflow/crates/otap/src/lib.rs	Exposes the new rate limiter and rate-limit middleware modules.
rust/otap-dataflow/crates/engine/tests/extension_e2e.rs	Updates engine extension E2E tests for new receiver factory capability field + pipeline build arg.
rust/otap-dataflow/crates/engine/src/lib.rs	Adds supported_rate_units to ReceiverFactory and enforces receiver/unit compatibility during receiver construction.
rust/otap-dataflow/crates/engine/src/config.rs	Extends ReceiverConfig to carry an optional resolved rate_limit policy into receivers.
rust/otap-dataflow/crates/core-nodes/src/receivers/traffic_generator/mod.rs	Declares no supported rate units for traffic generator receiver.
rust/otap-dataflow/crates/core-nodes/src/receivers/topic_receiver/mod.rs	Declares no supported rate units for topic receiver.
rust/otap-dataflow/crates/core-nodes/src/receivers/syslog_cef_receiver/README.md	Documents Syslog/CEF rate-limit semantics and adds new metrics/events.
rust/otap-dataflow/crates/core-nodes/src/receivers/syslog_cef_receiver/mod.rs	Integrates local rate limiting into Syslog/CEF UDP + TCP/TLS ingestion and adds metrics/tests.
rust/otap-dataflow/crates/core-nodes/src/receivers/otlp_receiver/README.md	Documents new OTLP rate-limit metrics.
rust/otap-dataflow/crates/core-nodes/src/receivers/otlp_receiver/mod.rs	Integrates rate limiting into OTLP receiver (gRPC + HTTP paths) and declares supported unit.
rust/otap-dataflow/crates/core-nodes/src/receivers/otap_receiver/mod.rs	Declares no supported rate units for OTAP receiver.
rust/otap-dataflow/crates/core-nodes/src/receivers/journald_receiver/mod.rs	Declares no supported rate units for journald receiver.
rust/otap-dataflow/crates/core-nodes/src/receivers/internal_telemetry_receiver/mod.rs	Declares no supported rate units for internal telemetry receiver.
rust/otap-dataflow/crates/core-nodes/src/receivers/host_metrics_receiver/mod.rs	Declares no supported rate units for host metrics receiver.
rust/otap-dataflow/crates/core-nodes/src/exporters/otlp_http_exporter/mod.rs	Updates OTLP HTTP exporter tests for the new server serve signature (rate limiter parameter).
rust/otap-dataflow/crates/controller/src/startup.rs	Validates receiver rate-limit capability at startup using supported_rate_units; updates engine validation to use resolved pipelines.
rust/otap-dataflow/crates/controller/src/live_control/runtime.rs	Threads resolved rate_limit policy into runtime pipeline construction.
rust/otap-dataflow/crates/controller/src/live_control/planning.rs	Rejects live reconciliation requests that would mutate the process-wide memory limiter config.
rust/otap-dataflow/crates/controller/src/live_control_tests.rs	Adds live-control test coverage for rejecting runtime memory limiter mutation; updates test receiver factories.
rust/otap-dataflow/crates/controller/src/lib.rs	Threads resolved rate_limit policy through controller pipeline build paths and adds semantic validation on controller start.
rust/otap-dataflow/crates/contrib-nodes/src/receivers/user_events_receiver/mod.rs	Declares no supported rate units for user events receiver.
rust/otap-dataflow/crates/contrib-nodes/src/receivers/etw_receiver/mod.rs	Declares no supported rate units for ETW receiver.
rust/otap-dataflow/crates/config/src/policy.rs	Adds rate_limit to policies + resolution + compatibility; defines RateLimitPolicy (including byte-unit parsing) and validation/tests.
rust/otap-dataflow/crates/config/src/engine/validate.rs	Adds semantic validation requiring a configured memory limiter when any rate_limit policy is present.
rust/otap-dataflow/crates/config/src/engine/resolve.rs	Ensures internal observability pipelines do not inherit user-facing rate_limit policy.
rust/otap-dataflow/crates/config/src/engine.rs	Adds policy resolution tests covering rate-limit override precedence and observability scoping.
rust/otap-dataflow/.chloggen/syslog-cef-rate-limit.yaml	Changelog entry for Syslog/CEF pressure-aware message-rate limiting.
rust/otap-dataflow/.chloggen/pressure-aware-rate-throttling.yaml	Changelog entry for OTLP pressure-aware rate throttling and the breaking ReceiverFactory embedding API change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated no new comments.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Status last refreshed: 2026-07-25 21:05:08 UTC.

• Waiting on: Author
• Next step: Address or respond to 1 review feedback item:
  • Inline threads: 1
  • For each item, reply to move the discussion forward, e.g. link to the commit that addresses it, explain why no change is needed, or ask a follow-up question.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected. If you believe this pull request is incorrectly routed as waiting on the author, comment /dashboard route:reviewers to route it from waiting on the author to waiting on reviewers. If the last refreshed time above predates your latest reply or push, the dashboard hasn't processed it yet.

[codecov]

Codecov Report

❌ Patch coverage is 92.13296% with 142 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.91%. Comparing base (34e6604) to head (0b370ad).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3529      +/-   ##
==========================================
+ Coverage   86.88%   86.91%   +0.03%     
==========================================
  Files         782      784       +2     
  Lines      319391   321151    +1760     
==========================================
+ Hits       277498   279136    +1638     
- Misses      41369    41491     +122     
  Partials      524      524              

Components	Coverage Δ
otap-dataflow	87.98% <92.13%> (+0.03%)	⬆️
query_engine	89.57% <ø> (ø)
otel-arrow-go	52.45% <ø> (ø)
quiver	92.20% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jmacd]

Feedback on configuration to align with #3583.

1. I was wondering if rate limits belong under resources. We have resources::memory_limiter already, so I think this should be rate_limiter; then, also I expect potentially multiple rate limits, so I think this should be rate_limiters.
2. Inside the rate limiter policy, I the allow, interval, and burst parameters are specific to a token bucket limiter, and I expect other kinds of limiter configuration in the policies framework, so I would indent these three under token_bucket.

groups:
  default:
    pipelines:
      main:
        policies:
          resources:
            rate_limiters:
              first_rate_limiter:
                unit: request_bytes/second
                token_bucket:
                  allow: 2000
                  interval: 1s
                  burst: 2000