From b83c7e5f2aa51f490ebea3cefb0b90a422658553 Mon Sep 17 00:00:00 2001 From: jaronoff97 Date: Thu, 2 Jul 2026 10:55:09 -0400 Subject: [PATCH 1/5] initial proposal --- projects/policies.md | 199 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 projects/policies.md diff --git a/projects/policies.md b/projects/policies.md new file mode 100644 index 000000000..38f13cbcf --- /dev/null +++ b/projects/policies.md @@ -0,0 +1,199 @@ +# Initial Policies Implementation + +## Background and description + +With the recent merge of the +[Policies OTEP](https://github.com/open-telemetry/opentelemetry-specification/pull/4738), +there is significant interest in implementing Policies across the OpenTelemetry +ecosystem. Policies introduce an intent-based specification for configuring how +telemetry components process data. A Policy is an atomic, self-contained rule +that declares a single intent — "match specific telemetry and apply an action" — +and can be evaluated identically regardless of where it runs. + +Because Policies are self-contained, independent, and can be evaluated in +parallel and scaled to thousands of rules without degradation. The same policy +behaves the same way whether it executes inside an SDK, a Collector, or any +other specification-compliant component, and policies are designed to be defined +centrally and distributed to running systems. + +This project coordinates the initial work needed to move Policies from an +accepted OTEP to an accepted specification: defining the specification, defining +the canonical Protobuf schemas, and building an initial prototype in the +Collector for validation. + +### Current challenges + +Configuring telemetry processing today is fragmented and component-specific. +Operators who want to filter, sample, redact, or transform telemetry must learn +a different configuration surface for each component — OTTL and processors in +the Collector, sampler configuration in each SDK, and vendor-specific mechanisms +elsewhere. A rule that drops debug logs or redacts PII has to be re-expressed, +re-tested, and re-maintained for every place it needs to run. + +Existing configuration problems: + +- **No portability.** A processing rule written for the Collector cannot be + moved closer to the source (into an SDK) without a rewrite, even though moving + it upstream is often the cheaper and safer place to enforce it. +- **No central management.** OpAMP exists to provide the capabilities for remote + management, however, without a consistent configuration method, there is no + way to centrally manage your telemetry infrastructure. +- **Ordering and coupling.** Existing pipeline configuration is order-dependent + and stateful, which makes rules hard to reason about in isolation and hard to + scale as their number grows. +- **Safety.** A malformed processing rule can break a pipeline and cause + telemetry loss. + +If nothing is done, users continue to hand-maintain overlapping, non-portable +configuration in every component, and the ecosystem lacks a common, governable +way to express telemetry-processing intent. + +### Goals, objectives, and requirements + +The aim of this project is to deliver an initial, working implementation of +Policies along with the specification and schemas that support it, proving the +intent-based model end to end in at least one component (the Collector) as a +foundation the rest of the ecosystem can build on. + +**Goals:** + +- Refine and stabilize the Policy specification so that it is general, minimal, + and implementable consistently across logs, metrics, and traces. +- Define the canonical Protobuf schema for Policies, with a human-authorable + YAML representation and bidirectional conversion (including shorthand forms). +- Build an initial prototype in the OpenTelemetry Collector that evaluates + policies through the two-stage model (match, then keep & transform) and honors + the core guarantees: atomicity, fail-open behavior, idempotence, and + data-model-based field references for portability. +- Establish match tracking (hits/misses) so operators can observe policy + effectiveness. +- Validate that the same policy semantics can later be implemented in SDKs. + +**Requirements the model must preserve:** + +- **Fail-open:** an invalid policy becomes inert rather than breaking valid + policies or dropping telemetry. +- **Atomic and self-contained:** one matcher set, one action set, no references + to other policies, no ordering dependency. +- **Portable:** field references use the OpenTelemetry data model so a policy + runs identically across conformant runtimes. + +**Why now:** the OTEP has just merged and there is broad interest across SIGs. +Coordinating the specification, schema, and a reference prototype now delivers +immediately on the goals of the OTEP. Delivering this gives OpenTelemetry a +common, governable way to express telemetry-processing intent that can be +enforced anywhere in the pipeline. + +## Deliverables + +- **Policy specification refinements.** Iterate on the merged OTEP into + specification text covering the two-stage execution model (match; keep & + transform), the matcher system (field selectors, match types, options, + `AttributePath` nested traversal), keep/sampling semantics for logs, metrics, + and traces, transform operations (remove, redact, rename, add), precedence + (most-restrictive-wins), and the error-handling taxonomy (parse, compilation, + runtime). +- **Canonical Protobuf schema** for Policies, plus the YAML authoring format and + documented bidirectional conversion including shorthand forms. +- **Collector prototype** implementing policy evaluation across logs, metrics, + and traces, including fail-open handling and match tracking. +- **Example policies** demonstrating transformation, sampling, and attribute + sanitization/redaction. +- **Integration direction** documenting how Policies relate to distribution + mechanisms (e.g. OpAMP, the OpenTelemetry Operator) and to policy composition + ("merger") — with dynamic distribution intentionally kept separate from core + policy semantics so both can iterate independently. + +Per OTEP requirements, specification work will be backed by working prototypes +in at least two languages/components before an OTEP for any follow-on refinement +is accepted. These requirements should be discussed with a TC member before +submitting. + +## Staffing / Help Wanted + +### Industry outreach (Optional) + +The Policies concept originated in discussions at KubeCon NA 2025 on +policy-based control of telemetry. Interested parties across the SDK, Collector, +Operator, and OpAMP SIGs, as well as vendors building telemetry-processing +products, should be aware of this effort and are invited to participate. + +### SIG + +A new Policies SIG will lead this work. Initial coordination and the reference +prototype will happen in collaboration with the Collector SIG. + +### Required staffing + +See [Project Staffing](/project-management.md#project-staffing) + +#### Project Leads(s) + +- **@jaronoff97** (Tero) – Maintainer +- **@jsuereth** (Google) – Maintainer +- **@dashpole** (Google) – Maintainer +- **@jackshirazi** (Elastic) – Maintainer + +#### Other Staffing + +Contributors from the Collector, SDK, Configuration, OpAMP, and Operator SIGs +are being recruited. Maintainers/approvers from those SIGs committed to +reviewing the specification and prototypes will be listed here as they commit. + +### Sponsorship + +See [Project Sponsorship](/project-management.md#project-sponsorship) + +#### TC Sponsor + +- @dashpole (Google) + +#### Delegated TC Sponsor (Optional) + +N/A + +#### GC Liaison + +TBD + +## Expected Timeline + +Relative to project start: + +- **Start:** form the Policies SIG, set up meetings and community table entries, + and align on the specification scope carried over from the OTEP. +- **~1 month in:** first draft of the canonical Protobuf schema and YAML + representation circulated for review. +- **~2 months in:** initial Collector prototype covering the match and keep + stages for logs, metrics, and traces. +- **~3–4 months in:** transform operations and match tracking in the prototype; + example policies published. +- **Following:** validate the model with a second implementation (SDK) toward + meeting the two-language prototype requirement, and document integration with + distribution mechanisms (OpAMP/Operator). + +After the project starts, specific target dates will be tracked via GitHub +project updates (see +[Project Lifecycle](project-management.md#project-lifecycle)). + +## Labels (Optional) + +- `policies` +- `spec:policies` +- `area:collector` + +## GitHub Project (Post-Approval) + +To be created from the +[GitHub Project template](https://github.com/orgs/open-telemetry/projects/140) +once the project is approved, pre-populated with issues covering the +deliverables above and organized by the timeline milestones. Link to be added +here. + +## SIG Meetings, Roadmap, and Other Info (Post-Approval) + +Meeting times, Slack channel, and meeting notes will be posted on the +[OpenTelemetry public calendar](https://github.com/open-telemetry/community#calendar) +and added to the community SIG tables via [workstreams.yml](./workstreams.yml) +(running `make generate`), including the GitHub project ID as a `roadmapProject` +entry so the project appears in the OpenTelemetry Roadmap. From 0dd339087869ce4c2d79aea753f3cfc32236bb79 Mon Sep 17 00:00:00 2001 From: jaronoff97 Date: Thu, 2 Jul 2026 10:56:29 -0400 Subject: [PATCH 2/5] add raphael --- projects/policies.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/policies.md b/projects/policies.md index 38f13cbcf..3524ab758 100644 --- a/projects/policies.md +++ b/projects/policies.md @@ -132,6 +132,7 @@ See [Project Staffing](/project-management.md#project-staffing) - **@jaronoff97** (Tero) – Maintainer - **@jsuereth** (Google) – Maintainer - **@dashpole** (Google) – Maintainer +- **@menderico** (Google) – Maintainer - **@jackshirazi** (Elastic) – Maintainer #### Other Staffing From 86126cfea1c2e0175ed9c2b3ad7dc6063bb30d68 Mon Sep 17 00:00:00 2001 From: jaronoff97 Date: Wed, 15 Jul 2026 11:47:15 -0400 Subject: [PATCH 3/5] update from feedback --- projects/policies.md | 138 ++++++++++++++++++++++++++----------------- 1 file changed, 85 insertions(+), 53 deletions(-) diff --git a/projects/policies.md b/projects/policies.md index 3524ab758..ac51c3b9e 100644 --- a/projects/policies.md +++ b/projects/policies.md @@ -10,16 +10,25 @@ telemetry components process data. A Policy is an atomic, self-contained rule that declares a single intent — "match specific telemetry and apply an action" — and can be evaluated identically regardless of where it runs. -Because Policies are self-contained, independent, and can be evaluated in -parallel and scaled to thousands of rules without degradation. The same policy -behaves the same way whether it executes inside an SDK, a Collector, or any -other specification-compliant component, and policies are designed to be defined +Policies are self-contained, independent, and can be evaluated in parallel and +scaled to thousands of rules without degradation. The same policy behaves the +same way whether it executes inside an SDK, a Collector, or any other +specification-compliant component, and policies are designed to be defined centrally and distributed to running systems. This project coordinates the initial work needed to move Policies from an -accepted OTEP to an accepted specification: defining the specification, defining -the canonical Protobuf schemas, and building an initial prototype in the -Collector for validation. +accepted OTEP into stable, usable policies people can rely on across +OpenTelemetry. Concretely, that means defining the data model and enforcement +specifications, defining the canonical Protobuf schemas, and shipping an **alpha +OpenTelemetry Collector component** that natively interprets and enforces +Policies. There are existing policy implementations already found in +[java-contrib](https://github.com/open-telemetry/opentelemetry-java-contrib) and +Tero's Policy work ([1](https://github.com/usetero/policy), +[2](https://github.com/usetero/policy-go), +[3](https://github.com/usetero/policy-zig)) that this project can build on as a +second implementation. "Done" for this project is a small set of stable policies +you can interact with; the components and features we need fall out from making +those policies successful. ### Current challenges @@ -57,17 +66,26 @@ foundation the rest of the ecosystem can build on. **Goals:** -- Refine and stabilize the Policy specification so that it is general, minimal, - and implementable consistently across logs, metrics, and traces. +- Deliver a small scoped set of stable policies — **sampling, filtering, and + transformation** — that work consistently across OpenTelemetry. Additional + policy types can be added iteratively once these are successful. +- Split and stabilize the specification into two documents: a **data model** + specification (the policy schema, matchers, and actions) and an + **enforcement** specification (how a component evaluates policies, including + the two-stage match then keep & transform model and the core guarantees). + Enforcment should also describe and require a set of conformance tests to + ensure that implementations are consistent. - Define the canonical Protobuf schema for Policies, with a human-authorable YAML representation and bidirectional conversion (including shorthand forms). -- Build an initial prototype in the OpenTelemetry Collector that evaluates - policies through the two-stage model (match, then keep & transform) and honors - the core guarantees: atomicity, fail-open behavior, idempotence, and - data-model-based field references for portability. -- Establish match tracking (hits/misses) so operators can observe policy - effectiveness. -- Validate that the same policy semantics can later be implemented in SDKs. +- Ship an alpha OpenTelemetry Collector component that **natively interprets and + enforces** policies (rather than translating them into existing Collector + configuration), honoring the core guarantees: atomicity, fail-open behavior, + idempotence, and data-model-based field references for portability. +- Use the host component's existing self-observability (e.g. Collector processor + self-obs metrics) to expose policy match tracking (hits/misses) so operators + can observe policy effectiveness, and scope out how a component reports + _which_ policies are applied (e.g. an OpAMP status tagged by policy + version/UUID, or self-observability signals for SDKs). **Requirements the model must preserve:** @@ -79,35 +97,43 @@ foundation the rest of the ecosystem can build on. runs identically across conformant runtimes. **Why now:** the OTEP has just merged and there is broad interest across SIGs. -Coordinating the specification, schema, and a reference prototype now delivers -immediately on the goals of the OTEP. Delivering this gives OpenTelemetry a -common, governable way to express telemetry-processing intent that can be -enforced anywhere in the pipeline. +Coordinating the specification, schema, and an alpha Collector component now +delivers immediately on the goals of the OTEP. Delivering this gives +OpenTelemetry a common, governable way to express telemetry-processing intent +that can be enforced anywhere in the pipeline. ## Deliverables -- **Policy specification refinements.** Iterate on the merged OTEP into - specification text covering the two-stage execution model (match; keep & - transform), the matcher system (field selectors, match types, options, - `AttributePath` nested traversal), keep/sampling semantics for logs, metrics, - and traces, transform operations (remove, redact, rename, add), precedence - (most-restrictive-wins), and the error-handling taxonomy (parse, compilation, - runtime). +- **Data model specification.** The policy schema itself: the matcher system + (field selectors, match types, options, `AttributePath` nested traversal), + actions, keep/sampling semantics for logs, metrics, and traces, and precedence + (most-restrictive-wins). +- **Enforcement specification.** How a component evaluates policies: the + two-stage execution model (match; then keep & transform), the core guarantees + (atomicity, fail-open, idempotence, portability), and the error-handling + taxonomy (parse, compilation, runtime). - **Canonical Protobuf schema** for Policies, plus the YAML authoring format and documented bidirectional conversion including shorthand forms. -- **Collector prototype** implementing policy evaluation across logs, metrics, - and traces, including fail-open handling and match tracking. -- **Example policies** demonstrating transformation, sampling, and attribute - sanitization/redaction. +- **Alpha Collector component** that natively interprets and enforces policies + across logs, metrics, and traces (not a translation into existing Collector + configuration), including fail-open handling and match tracking. +- **Experimental policies** for the initial scope: **sampling, filtering, and + attribute sanitization**. These are the crisp deliverables that let us prove + the model quickly; transformation and other policy types are deferred and + added iteratively. +- **Applied-policy reporting.** Scope and formalize how a component reports + which policies are actually in effect — for example an OpAMP status tagged by + policy version/UUID, or self-observability signals for SDKs — so operators can + see what is applied to a given instance. - **Integration direction** documenting how Policies relate to distribution mechanisms (e.g. OpAMP, the OpenTelemetry Operator) and to policy composition ("merger") — with dynamic distribution intentionally kept separate from core policy semantics so both can iterate independently. -Per OTEP requirements, specification work will be backed by working prototypes -in at least two languages/components before an OTEP for any follow-on refinement -is accepted. These requirements should be discussed with a TC member before -submitting. +Following OpenTelemetry specification practice, the alpha component(s) — the +Collector component here, plus the existing java-contrib implementation as a +second — come before the specification is formalized. These requirements should +be discussed with a TC member before submitting any follow-on OTEP. ## Staffing / Help Wanted @@ -120,8 +146,8 @@ products, should be aware of this effort and are invited to participate. ### SIG -A new Policies SIG will lead this work. Initial coordination and the reference -prototype will happen in collaboration with the Collector SIG. +A new Policies SIG will lead this work. Initial coordination and the alpha +Collector component will happen in collaboration with the Collector SIG. ### Required staffing @@ -129,17 +155,20 @@ See [Project Staffing](/project-management.md#project-staffing) #### Project Leads(s) -- **@jaronoff97** (Tero) – Maintainer -- **@jsuereth** (Google) – Maintainer -- **@dashpole** (Google) – Maintainer -- **@menderico** (Google) – Maintainer -- **@jackshirazi** (Elastic) – Maintainer +- **@jaronoff97** (Tero) – Maintainer; Operator, Helm, Injector +- **@jsuereth** (Google) – Maintainer; Specification, Collector +- **@dashpole** (Google) – Maintainer; Specification, Collector, Prometheus +- **@menderico** (Google) – Maintainer; Collector +- **@jackshirazi** (Elastic) – Maintainer; Java SDK/agent, author of the + existing policy implementation in + [java-contrib](https://github.com/open-telemetry/opentelemetry-java-contrib) #### Other Staffing Contributors from the Collector, SDK, Configuration, OpAMP, and Operator SIGs are being recruited. Maintainers/approvers from those SIGs committed to -reviewing the specification and prototypes will be listed here as they commit. +reviewing the specification and alpha components will be listed here as they +commit. ### Sponsorship @@ -162,19 +191,22 @@ TBD Relative to project start: - **Start:** form the Policies SIG, set up meetings and community table entries, - and align on the specification scope carried over from the OTEP. + and lock the initial policy scope (sampling, filtering, attribute + sanitization). - **~1 month in:** first draft of the canonical Protobuf schema and YAML - representation circulated for review. -- **~2 months in:** initial Collector prototype covering the match and keep - stages for logs, metrics, and traces. -- **~3–4 months in:** transform operations and match tracking in the prototype; - example policies published. -- **Following:** validate the model with a second implementation (SDK) toward - meeting the two-language prototype requirement, and document integration with + representation, and the data model / enforcement specification split, + circulated for review. +- **~2 months in:** alpha Collector component enforcing the initial policies, + covering the match and keep stages for logs, metrics, and traces. +- **~3–4 months in:** match tracking and applied-policy reporting in the alpha + component; experimental policies published. +- **Following:** align the java-contrib implementation as a second + implementation of the same policies, and document integration with distribution mechanisms (OpAMP/Operator). -After the project starts, specific target dates will be tracked via GitHub -project updates (see +The SIG will give regular (roughly monthly) updates to the Maintainer and +Specification SIGs. After the project starts, specific target dates will be +tracked via GitHub project updates (see [Project Lifecycle](project-management.md#project-lifecycle)). ## Labels (Optional) From 36f206cb0bb662d75480c2969cb4156e3ee1a57f Mon Sep 17 00:00:00 2001 From: jaronoff97 Date: Wed, 15 Jul 2026 11:50:56 -0400 Subject: [PATCH 4/5] lang --- projects/policies.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/projects/policies.md b/projects/policies.md index ac51c3b9e..bd4b117f7 100644 --- a/projects/policies.md +++ b/projects/policies.md @@ -67,8 +67,9 @@ foundation the rest of the ecosystem can build on. **Goals:** - Deliver a small scoped set of stable policies — **sampling, filtering, and - transformation** — that work consistently across OpenTelemetry. Additional - policy types can be added iteratively once these are successful. + scoped transformations** — that work consistently across OpenTelemetry. More + transform types and additional policy types can be added iteratively once + these are successful. - Split and stabilize the specification into two documents: a **data model** specification (the policy schema, matchers, and actions) and an **enforcement** specification (how a component evaluates policies, including @@ -118,9 +119,8 @@ that can be enforced anywhere in the pipeline. across logs, metrics, and traces (not a translation into existing Collector configuration), including fail-open handling and match tracking. - **Experimental policies** for the initial scope: **sampling, filtering, and - attribute sanitization**. These are the crisp deliverables that let us prove - the model quickly; transformation and other policy types are deferred and - added iteratively. + scoped transformations**. Additional transform types for things like + datapoints and other policy types are deferred and added iteratively. - **Applied-policy reporting.** Scope and formalize how a component reports which policies are actually in effect — for example an OpAMP status tagged by policy version/UUID, or self-observability signals for SDKs — so operators can From d2141c85e125f470900cf4f133c3eb808fdea020 Mon Sep 17 00:00:00 2001 From: jaronoff97 Date: Mon, 20 Jul 2026 10:47:52 -0400 Subject: [PATCH 5/5] feedback --- projects/policies.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/projects/policies.md b/projects/policies.md index bd4b117f7..2d6ee5e01 100644 --- a/projects/policies.md +++ b/projects/policies.md @@ -93,7 +93,10 @@ foundation the rest of the ecosystem can build on. - **Fail-open:** an invalid policy becomes inert rather than breaking valid policies or dropping telemetry. - **Atomic and self-contained:** one matcher set, one action set, no references - to other policies, no ordering dependency. + to other policies, no ordering dependency. This means no policy depends on + another policy having already been applied — it does not mean policies + cannot conflict or overwrite each other's effects; precedence rules (see + Deliverables) govern that. - **Portable:** field references use the OpenTelemetry data model so a policy runs identically across conformant runtimes. @@ -207,7 +210,7 @@ Relative to project start: The SIG will give regular (roughly monthly) updates to the Maintainer and Specification SIGs. After the project starts, specific target dates will be tracked via GitHub project updates (see -[Project Lifecycle](project-management.md#project-lifecycle)). +[Project Lifecycle](/project-management.md#project-lifecycle)). ## Labels (Optional) @@ -227,6 +230,6 @@ here. Meeting times, Slack channel, and meeting notes will be posted on the [OpenTelemetry public calendar](https://github.com/open-telemetry/community#calendar) -and added to the community SIG tables via [workstreams.yml](./workstreams.yml) +and added to the community SIG tables via [workstreams.yml](/workstreams.yml) (running `make generate`), including the GitHub project ID as a `roadmapProject` entry so the project appears in the OpenTelemetry Roadmap.