Add executor thread pool metrics instrumentation

[YaoYingLong]

Relates to #11149

Motivation

Executor metrics provide visibility into utilization, saturation, queue pressure, completed work, and rejected work. This change adds an initial implementation to the existing Java executors instrumentation and makes the proposed metric model concrete for semantic-convention discussion.

Design and lifecycle

ThreadPoolExecutor instances are pre-registered at construction time, but no metrics are created and the configured ThreadFactory is not invoked then. Metrics are registered only after the first worker thread actually starts, using that worker's real thread name to derive jvm.executor.name.

When setThreadFactory() is called, the existing metric identity remains active until a worker created by the new factory starts. At that point, the registration is replaced using the new worker thread name.

Generic JDK instrumentation uses jvm.executor.owner.name=unknown. Framework-specific instrumentation can reuse reregister(...) to provide an owner name and a component-specific all or trailing name-normalization mode. This works both before the first worker starts and after metrics are already registered.

Rejected tasks are recorded after metrics have been registered. Metrics are unregistered only after an executor has actually entered the shutdown state. The registration cache and callbacks use weak references, so metrics do not retain an unclosed executor; callbacks close themselves after the observed executor is collected.

Metrics emitted by ThreadPoolExecutor

Metric	Instrument	Unit	Attributes
jvm.executor.thread.count	Observable UpDownCounter	{thread}	jvm.executor.name, jvm.executor.owner.name, jvm.executor.type, jvm.executor.state
jvm.executor.thread.core	Observable UpDownCounter	{thread}	jvm.executor.name, jvm.executor.owner.name, jvm.executor.type
jvm.executor.thread.max	Observable UpDownCounter	{thread}	jvm.executor.name, jvm.executor.owner.name, jvm.executor.type
jvm.executor.queue.size	Observable UpDownCounter	{task}	jvm.executor.name, jvm.executor.owner.name, jvm.executor.type
jvm.executor.queue.remaining	Observable UpDownCounter	{task}	jvm.executor.name, jvm.executor.owner.name, jvm.executor.type
jvm.executor.task.completed	Observable Counter	{task}	jvm.executor.name, jvm.executor.owner.name, jvm.executor.type
jvm.executor.task.rejected	Counter	{task}	jvm.executor.name, jvm.executor.owner.name, jvm.executor.type

JDK 21 ThreadPerTaskExecutor is also covered. It currently emits only jvm.executor.thread.count with jvm.executor.state=active.

jvm.executor.name is derived from the first actual worker thread name. otel.instrumentation.executors.name-normalization=all is the default and replaces every digit group with *; trailing replaces only the final digit group. For example, tomcat-8080-exec-1 becomes tomcat-*-exec-* with all and tomcat-8080-exec-* with trailing. Empty thread names use unknown.

Scope and follow-up

This is an initial implementation intended to support discussion of experimental executor metric semantic conventions. ScheduledThreadPoolExecutor and jvm.executor.queue.wait.duration are intentionally out of scope. Once the metric and attribute semantics are agreed, the reusable registration path can evolve into an incubating shared helper for framework-specific instrumentations.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Status last refreshed: 2026-07-24 05:19:52 UTC.

• Waiting on: Reviewers
• Next step: Review the latest changes.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected.

[Copilot]

Pull request overview

Adds automatic executor metrics for JDK thread pools, including reusable registration for future framework-specific instrumentation.

Changes:

• Adds executor metric instruments and weak-reference registrations.
• Instruments ThreadPoolExecutor and JDK 21 ThreadPerTaskExecutor lifecycles.
• Adds name-normalization configuration, documentation, and tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
instrumentation/executors/README.md	Documents name normalization.
instrumentation/executors/metadata.yaml	Defines the new configuration.
instrumentation/executors/jdk21-testing/.../ThreadPerTaskExecutorMetricsTest.java	Tests JDK 21 executor metrics.
instrumentation/executors/javaagent/.../ThreadPoolExecutorMetricsTest.java	Tests thread-pool metrics and lifecycle behavior.
instrumentation/executors/javaagent/.../ThreadPoolExecutorMetricsInstrumentation.java	Instruments thread-pool lifecycle methods.
instrumentation/executors/javaagent/.../ThreadPerTaskExecutorMetricsInstrumentation.java	Instruments JDK 21 per-task executors.
instrumentation/executors/javaagent/.../ExecutorsInstrumentationModule.java	Registers the new instrumentations.
instrumentation/executors/javaagent/build.gradle.kts	Adds trailing-normalization tests.
instrumentation/executors/bootstrap/.../ThreadPoolExecutorMetrics.java	Implements registration and metric callbacks.
instrumentation/executors/bootstrap/.../JvmExecutorMetrics.java	Defines executor metric instruments.
instrumentation/executors/bootstrap/build.gradle.kts	Adds required bootstrap dependency.

[jaydeluca]

@trask should there be a corresponding semconv issue for this?

[jaydeluca]

The code that uses these names to check if the module is enabled/disabed as part of AgentDistributionConfig.isInstrumentationEnabled uses the first record it finds, so If someone has set otel.instrumentation.executors.enabled=true it will also enable this module by default, and ignore the defaultEnabled value, even if otel.instrumentation.executors-metrics.enabled is not set

[YaoYingLong]

Initially, I planned to follow the implementation approach used in the Kafka instrumentation.

Reference code link: KafkaMetricsInstrumentationModule

For example, I intended to declare it via the super constructor call: super("executors-metrics", "executors").

However, the check-javaagent-suppression-keys.sh validation failed during execution.

After investigation, extra handling logic needs to be added into this shell script.

Nevertheless, there is an existing TODO comment inside the script, leaving me uncertain about the optimal modification strategy.

Two options are available: either extract the relevant logic into a standalone module, or directly edit the check-javaagent-suppression-keys.sh script to make the test pass. I would like to discuss this matter with the team to reach a consensus.

[jaydeluca]

perhaps this should be consistent with the other one?

Suggested change

	DeclarativeConfigUtil.getInstrumentationConfig(GlobalOpenTelemetry.get(), "executors")
	DeclarativeConfigUtil.getInstrumentationConfig(GlobalOpenTelemetry.get(), "executors-metrics")

[YaoYingLong]

This issue has been considered beforehand. If we rename executors to executors-metrics, do we also need to change io.opentelemetry.executors to io.opentelemetry.executors-metrics?

I referenced the existing Kafka implementation cases, so for now we keep our logic consistent with Kafka's implementation.

KafkaMetricsUtil

[jaydeluca]

this looks like it is only used in tests right now. I'm guessing maybe its part of the rest of the work mentioned in the PR description? as of now ownerName is only ever mutated by reregister, so it will always be "unknown"

[YaoYingLong]

Currently this field is hardcoded to unknown. When other components adopt the JDK-native ThreadPoolExecutor and ThreadPerTaskExecutor thread pools (which we predefine) in the future, we can reset this field to mark the corresponding component being used.

Reference examples:

https://github.com/jetty/jetty.project/blob/ce3c61a30e84425ae11772ab4f61aa2ee434d69d/jetty-core/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L93

https://github.com/apache/tomcat/blob/cb9c4bce797b671a6ba453360e24e00b3622f3e1/java/org/apache/tomcat/util/net/AbstractEndpoint.java#L1071

I have finished implementing this part. I will submit a new PR only after the semantic specifications for thread pool metrics are finalized and the current PR gets merged.