[OTEP] Federate Semantic Conventions

[jsuereth]

This is a proposal to federate the semantic-conventions repository to allow for faster innovation and evolution across the ecosystem.

• We retain clear ownership of semantic convention domains
• We retain clear policy and tooling support for instrumentation authors to provide comprehensive documentation, stability guarantees and integration test assurance of the signals the produce.
• We provide a clear "shared" location for x-cutting attributes and definitions.
• We provide a path for federated semantic conventions to promote to core, if desired.
• We expand for non-opentelemetry sources of conventions and standards to advertise their instrumentation and leverage the same tooling as open-telemetry.

[lmolkova]

so that there is no chance of having conflicts / vendoring in in incompatible manner / etc?

I like it!

[lmolkova]

this is awesome!

[lmolkova]

nit: maybe explain what policies are (link to some section/doc), I imagine it's not a commonly known concept

[lmolkova]

nit: it's the same on as the last one? (policies/check/naming)

[lmolkova]

nit:

I believe we now have the opt-in mechanism that's slightly different than picking a schema url, it was just introduced in declarative config: https://github.com/open-telemetry/opentelemetry-configuration/blob/4351ebd2805746d047a23588f8d3abe89f40f79f/snippets/ExperimentalInstrumentation_kitchen_sink.yaml#L52

E.g.

instrumentation:
  general:
    rpc:
      semconv:
        version: 1
        experimental: false
        dual_emit: true

[lmolkova]

Do we want to allow stable (federated) registry to depend on unstable parts of otel conventions?

I think we can go either way.
If we don't allow it: they can take dependency on stable parts only and vendor in unstable parts if necessary.
If we allow it, they should be able to communicate breaking changes in unstable core via semver.

Either way, we need some forcing factors for them to

• update to newer underlying core - it will be hard regardless
• bring common concepts to core conventions

I'd rather not allow it at least initially and encourage to vendor in experimental stuff.

[lmolkova]

this would probably be the forcing factor to update federated registries I mentined in prev comment.
If library A depends on semconv core v1.40 and library B on semconv core v1.140 they probably can't coherently work in the same distro.

[lmolkova]

nit: would we separate HTTP from core conventions? Probably not until it needs v2

[lmolkova]

This is awesome!

I also like the year as a major version - we should create expectation of some breaking changes on a predictable cadence

[mikeldking]

This is date ver no? This is not semver.

[mikeldking]

How does this propagate to natively instrumented libraries and what incentive do they have to move to newer conventions? There always is a transform layer running?

[jmacd]

As an alternative to a transform layer running always, I wish to see more-flexible ways to reconfigure static transforms over original instrumentation. In other words, to modify the telemetry at the start instead of in a pipeline (ideally using code generated from the semantic conventions).

[mikeldking]

are conventions in a particular version all uniformly considered "stable"?

[mikeldking]

what are the rules of semver here. Like dropping an attribute is "breaking" - what if it was optional?

[kamphaus]

Removing an opt-in attribute would still be breaking for those users that have opted-in.

[mikeldking]

What's the acceptance criteria for "tested together", what if an instrumentation fails, is it removed from release?

[mikeldking]

What if end-users want to opt out of expensive / PII risking conventions. Do these still land in core? What is the final say for something deserving to be in core. I'm unfamiliar here.

[jsuereth]

We allow expensive / PII in core, with opt_in today and there's guidance around it. I think this is an orthogonal concern to this proposal, but an important issue that needs a solution across various components of OpenTelemetry.

For this proposal, we continue with the current behavior we have.

[mikeldking]

this is a good example, but also centralized with the maintainers of Java. So it seems like a simpler example? It's harder to do this when there is no central vendor.

[mikeldking]

where to these actions run - on OTEL repos only? How does it run on native libraries.

[jsuereth]

So we only "enforce" this structure for OTEL-owned repositories/conventions.

For native libraries, we provide open-telemetry/weaver as a tool they can use to participate in this federation, but we do not enforce any structure / policy unless they are planning to move their definitions back into the OpenTelemetry project.

[mikeldking]

what if the federated conventions need the unstable conventions for it to make sense - example being session / threading of agents.

[jsuereth]

Then the federated convention would be marked "unstable" as well. Basically you can't declare a "child" convention stable unless the "parent" is also stable.

This doesn't preclude usage, just stability declaration.

[mikeldking]

dual_emit prioritizes which verion?

[jsuereth]

In OpenTelemetry - dual_emit will fire both conventions, i.e. you'll get both attributes, multiple metrics, etc.

@trask or @lmolkova may be able to speak more to how that works in practice.

[Aneurysm9]

What does "both conventions" mean, though? Some prior version of the jvm federated registry? The vestigial conventions in the semconv registry?

[mikeldking]

which users are we talking about. The end-user?

[jsuereth]

Yes

[mikeldking]

this is not so simple. Instrumentation is dependent on now two versions, the library it instruments AND the semcov that it tracks. What if the instrumentation needs to move a major version for library changes, then subsequently needs to backfill support for an older version. Now there is no tenable way to track both semcov breaking changes with library breaking changes.

[jsuereth]

I think I intend this to mean the opposite, or at least I don't see them as coupled.

• If you make a breaking change in semconv, you need a major version bump on your library.
• You may need major version bump on your library for other reasons, you do NOT need to bump semconv major version for this.

The key here is folks understand they need to do the first, and that the output signals of an instrumentation library are part of its stability.

Regarding tracking both semconv breaking + library breaking changes - I wanted to keep this simpler where you mostly just pay attention to library major version bumps, and then schema_url and backends can help you handle incompatibilities / conversions downstream.

[kamphaus]

then subsequently needs to backfill support for an older version

@mikeldking Are you asking about backporting a semconv breaking change to an older instrumentation version?

[mikeldking]

What if the library is unstable?

[jsuereth]

We would allow semconv to be marked as stable prior to libraries being fully stable. This is more to prevent the opposite in otel - libraries which are production-ready NOT declaring stability, because of a looming threat of semconv changes.

[mikeldking]

the problem here is "cost" - the cost to one OTLP backend might be significant - it dictates a business deal being made in a quarter vs losing to a competitor that doesn't comply to Otel at all. This puts vendors that are all in on OTEL at risk as their upstream signals are being dictated by parties that have no actual financial overhead or incentives to move the standards forward.

[Aneurysm9]

I'm also not sure this aligns with the Contrib Stability section of the versioning and stability doc:

Plugins, instrumentation, and other contrib packages SHOULD be kept up to date and compatible with the latest versions of the API, SDK, and Semantic Conventions. If a release of the API, SDK, or Semantic Conventions contains changes which are relevant to a contrib package, that package SHOULD be updated and released in a timely fashion.

If the relevant federated semconv decides to ship new major versions every other Tuesday because they're post-1.x and yet have attributes or telemetry that are in flux then it seems like the instrumentation libraries that depend on it would also be expected to ship new major versions frequently. Maybe that's something we'd want the TC to put a stop to, but it does seem like an expected result of a combined push for "stable semver" and "faster evolution".

[jsuereth]

So the idea behind this federation is that the lifecycle of a federated semconv and the instrumentation are aligned. You wouldn't bump one without the other, but you can bump both without bumping core semconv versions or requiring syncing beyond that scope.

The owners of the instrumentation and the owners of the "federated semconv" would be the same set of folks.

The wording in the Contrib Stability section would be updated appropriately here as well.

[mikeldking]

how can we say that backends will use these agent-aided transformations? That seems like something that certain vendors such as companies that have privileged foundation model access a competitive advantage. This seems unfair.

[jsuereth]

Good point, and not the real intention here.

The agent-aided meant we may want to start leveraging agents within open-telemetry to define transitions at definition time, and publicizing these transitions for everyoen to consume without agents. We cannot have agent-in-the-loop on hot paths, that's intractable. Leveraging an agent to help improve the coverage of how many version transitions we can safely provide is what this was intended to mean.

The true goal is that we have an open model and easy-to-use transition "definition" that folks can consume to translate between versions of schema, or even (if we're successful) between different schema-urls entirely.

That's still experimental / long-term exploration for the project. For now, we have targeted version-bump transitions as something we believe we can provide automatic transition model that should NOT be costly unfair for the ecosystem to adopt.

[mikeldking]

I'm not familiar with the weaver MCP server. How does this work?

[jsuereth]

Today it's pretty light. You run weaver mcp and it will load in a registry and allow the agent to interact with a version of semantics and lookup the model, definitions, notes, etc. It can also run live-check to enforce conformance on the version. We hope to expand this further to be more of an aide when defining conventions and allow more ad-hoc agent-initiated flows, e.g. "Help me create a semantic convention registry from this integration test"

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[dashpole]

It would also be nice to think through where the code generators and generated code would live. Would the existing language generators also track and generate federated repos, or would they need to publish their generation artifacts (e.g. templates) in a way that is consumable (and stable?) for federated semconv publishers to use to generate their own stuff? Presumably we don't want federated publishers to also have to write their own code generators.

[jsuereth]

Code generators - https://github.com/open-telemetry/opentelemetry-weaver-packages

Generated code - perhaps this needs more though. My expectation was it would live beside the instrumentation that exists today. I think that should be sufficient.

[dashpole]

Got it. So stuff like https://github.com/open-telemetry/opentelemetry-go/tree/main/semconv/templates/registry/go would need to be replicated in each federated semconv area? Or is go the only ones that have our templates stored in our repo?

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[Aneurysm9]

What does "both conventions" mean, though? Some prior version of the jvm federated registry? The vestigial conventions in the semconv registry?

[Aneurysm9]

This feels like the opposite of how I understood the proposal in #4813. That seemed to say that instrumentation libraries should declare themselves stable without regard to the stability of the semantic conventions they use. This seems to be saying that semantic conventions used by stable instrumentation libraries must be, by definition, stable. What of an instrumentation library that is "stable" but for dependence on semantic conventions that are "experimental"?

[jsuereth]

You can see my back and forth about #4813 - That was the wrong impression and the wording (should be corrected).

The goal is that insrumetnation can declare stability prior to core semconv declaring stability, but that instrumentation must then maintain its own stability of what it produces. This is the mechanism for how.

[Aneurysm9]

Does making major version bumps when required count as remaining backward compatible? If not I'd fear that such a time-in-grade requirement could lead to intentional stagnation.

[jsuereth]

I'm not sure I parse what you are asking.

[Aneurysm9]

If the jvm conventions have been stable for 6mo and want to be merged into core, but recognize a need for a breaking change can they do so with a MV bump without resetting the clock on that 1y timer? Put another way, does "Reach stable status, and remain backwards compatible for a full year" mean the highest MV must be >1 and at least one year old or does it mean that v1.0 must be at least one year old and any breaking changes appropriately incremented the MV? If the former, would they be incentivized to sit on a change for 6mo to get v1.x into core before releasing v2.x? Is that the incentive we want?

[Aneurysm9]

I'm also not sure this aligns with the Contrib Stability section of the versioning and stability doc:

Plugins, instrumentation, and other contrib packages SHOULD be kept up to date and compatible with the latest versions of the API, SDK, and Semantic Conventions. If a release of the API, SDK, or Semantic Conventions contains changes which are relevant to a contrib package, that package SHOULD be updated and released in a timely fashion.

If the relevant federated semconv decides to ship new major versions every other Tuesday because they're post-1.x and yet have attributes or telemetry that are in flux then it seems like the instrumentation libraries that depend on it would also be expected to ship new major versions frequently. Maybe that's something we'd want the TC to put a stop to, but it does seem like an expected result of a combined push for "stable semver" and "faster evolution".

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[kamphaus]

Can / should the checks use versioned opentelemetry-weaver-packages?

[jsuereth]

Absolutely! We need to cut our first release of that repo

[kamphaus]

then subsequently needs to backfill support for an older version

@mikeldking Are you asking about backporting a semconv breaking change to an older instrumentation version?

[kamphaus]

How would we communicate that the federated semconv has merged into core?
How would we ensure that users can discover that further versions are available in core?
Would the federated manifests keep getting new versions for every core semconv version or would it include a flag moved_to?
For sure its repo could show a message in the README and a message in its last release before being archived.
Is this sufficient or should new semconv version in core be programmatically discoverable?

[kamphaus]

How would federated registries not hosted under opentelemetry interact with the platform release?
Eg a corporate registry declares dependencies on the core otel and jvm registries, could the corporate registry use the platform release manifest as a dependency or would they need to update their own registry dependency manifest to use the platform release versions?

[kamphaus]

Removing an opt-in attribute would still be breaking for those users that have opted-in.

[lmolkova]

LGTM, this is effectively being done in GenAI SIG https://github.com/open-telemetry/semantic-conventions-genai/ which can serve as a prototype

[lmolkova]

nit: do you think we have to have a new type of manifest (platform)?

This is essentially publication manifest with import * from all dependencies.
I imagine the core difference is that this is never published as a registry and does not have schema url

[lmolkova]

we would essentially own platform release in semconv. The platform release will also ensure that there are no collisions between otel-hosted registries

[lmolkova]

One thing we didn't consider yet is semconv libs generation: whether there should be a semconv lib per federated registry and/or one lib for all applicable registries.

I think a viable approach for language SIGs to have one user-facing lib with a curated list of registries applicable to their ecosystem (e.g. JVM metrics are only important for Java, maybe Kotlin and Android ecosystems but not for Python).

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[jmacd]

As an alternative to a transform layer running always, I wish to see more-flexible ways to reconfigure static transforms over original instrumentation. In other words, to modify the telemetry at the start instead of in a pipeline (ideally using code generated from the semantic conventions).

[MrAlias]

Thanks for writing this up. I think federation is a useful direction, but from the Go semantic-convention implementation side I do not think this OTEP yet gives language maintainers enough contract to implement it cleanly.

Today Go has a simple generated-package model: users choose a semantic convention version and import a versioned package such as go.opentelemetry.io/otel/semconv/v1.42.0. That package is generated from the corresponding core semantic-conventions release and contains helpers for HTTP, RPC, messaging, database, system, process, resources, and so on. If GenAI, MCP, OpenAI, JVM, or future domains move into independent release streams with independent schema URLs, Go needs to know whether those become separate generated packages, one curated language package, a platform-release package, or something else.

The sharpest blocker is the schema URL model for mixed telemetry. Existing Go instrumentation already emits attributes from multiple semantic-convention namespaces in one instrumentation scope. HTTP instrumentation emits http.*, url.*, server.*, network.*, and error.type; AWS SDK v2 emits rpc.*, aws.*, messaging.*, db.*, and server.*; MongoDB emits db.* and network.*; gRPC metrics combine RPC conventions with server.* attributes and compatibility modes. Resource detectors are even more mixed: AWS ECS, GCP, Azure VM, and the built-in Go SDK resource providers combine namespaces such as cloud.*, container.*, k8s.*, faas.*, host.*, os.*, service.*, process.*, and telemetry.sdk.*.

That all works today because the active conventions are covered by one core schema URL, or in several contrib instrumentations no schema URL is set on the scope at all. Under federation, it is not clear whether the schema URL for mixed telemetry should be the owning registry for the span/metric/log, a resolved closure of that registry plus dependencies, a platform release, or blank/schemaless on conflict. This needs to be specified before we can tell Go users which generated package to import or which SchemaURL instrumentation should use.

Before approving this, I think the OTEP needs to make the following explicit: how shared cross-cutting attributes such as server.*, network.*, error.type, cloud.*, host.*, service.*, process.*, and os.* are owned; how dependency schema URLs are resolved when attributes from different registries appear on the same scope or resource; how diamond dependencies map to language artifacts where latest wins is not how imports work; how resources with attributes from different schema URLs are represented; and how language packages, generated code, discovery, and version pairing are expected to work for GenAI and future federated registries.

I am not opposed to federation, but without those answers the OTEP leaves the hardest interoperability questions to language SIGs and instrumentation authors to infer independently.

[MrAlias]

I think this needs to define what dependency means for schema URLs and generated language artifacts. If a federated registry depends on core semantic conventions, does its schema URL identify only the local registry, or a resolved closure that also covers the dependency attributes it uses?

For Go this is not just a manifest detail. Generated semconv packages are versioned import paths, so a future genai/v0.x.0 package and semconv/v1.42.0 package would expose separate constants and helpers. The OTEP should say whether instrumentation using a federated registry should use the federated SchemaURL, the core SchemaURL, a platform-release schema URL, or something else when it emits attributes from both.

[MrAlias]

This is underspecified for instrumentation that emits mixed semantic-convention namespaces from one scope. Existing Go contrib instrumentation already does this: HTTP emits http.*, url.*, server.*, network.*, and error.type; AWS SDK v2 emits rpc.*, aws.*, messaging.*, db.*, and server.*; MongoDB emits db.* and network.*; gRPC metrics combine RPC conventions with server.* attributes.

Since a Tracer/Meter/Logger scope has one schema URL, the OTEP needs to say which schema URL such instrumentation should set. If the answer is the federated registry's schema URL, then that URL needs to be defined as a resolved schema that also covers dependency attributes. If not, language maintainers need the alternative rule.

[MrAlias]

This unique-namespace rule does not explain shared cross-cutting conventions. Namespaces such as server, network, error, url, cloud, host, service, process, and os are not naturally tied to one span/metric/log domain, and current instrumentation and resource detectors reuse them across domains.

If the intended atom is the span, metric, or log, where do these shared attribute groups live? If they remain core, the OTEP should say that federated registries depend on core for these groups. If they can themselves federate, the OTEP needs a rule for instrumentation that combines a domain registry with one or more shared registries.

[MrAlias]

The reexport story needs to be more precise for language artifacts and user discovery. If gen_ai.* starts in a federated Go package and later moves into core semconv/v1.N.0, what happens to the old package's SchemaURL, generated helpers, documentation, and import path? Should the old federated package keep releasing wrappers, expose a moved_to signal, or stop at the last federated version?

Without a programmatic migration/discovery rule, users will not know whether to keep importing the federated package or move to the core semconv package after promotion.

[MrAlias]

latest version wins is not enough of a resolution rule for language implementations. In Go, semantic convention versions are import paths, not dependency constraints that a solver collapses to one version. A program can import go.opentelemetry.io/otel/semconv/v1.42.0, another package can import v1.43.0, and a future GenAI package might depend on a particular core semconv version.

The OTEP should define what language maintainers expose when a federated registry depends on core 1.42.0 but the rest of an application or instrumentation set uses core 1.43.0. The answer may be a resolved federated schema, a platform-release artifact, or separate package constants, but it needs to be explicit.

[MrAlias]

I do not think the implementation burden is optional for language SIGs. Even if SDKs do not perform schema transformations, language maintainers still need to decide how generated semconv packages are published, how users discover which package owns a namespace, how package versions pair with core versions, and which SchemaURL constants instrumentation should use for mixed telemetry.

This should be part of the OTEP's compatibility contract rather than left as language-specific interpretation.

[MrAlias]

This should also cover resources, not only InstrumentationScope. Existing Go resource detectors commonly create one resource containing attributes from multiple namespaces: AWS ECS combines cloud.*, container.*, AWS ECS, and AWS log attributes; GCP combines cloud.*, k8s.*, faas.*, host.*, and GCP attributes; Azure VM combines cloud.*, host.*, and os.*; built-in Go SDK providers cover service.*, telemetry.sdk.*, host.*, process.*, os.*, and container.*.

The Go SDK already treats different non-empty resource schema URLs as a merge conflict and returns a schemaless merged resource. If federation can assign different schema URLs to those attribute groups, the OTEP needs to say whether resources use a resolved schema URL, a platform schema URL, no schema URL, or some other rule.

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.