wildfly-per-deployment-instrumentation

A reference implementation and guide for enabling per-deployment OpenTelemetry service.name on a WildFly application server, replacing a JVM-level Splunk Java agent with WildFly's built-in OpenTelemetry subsystem.

Why this exists

When multiple applications are deployed to the same WildFly JVM and instrumented with the Splunk Java agent, all spans share a single service.name (e.g. wildfly). This makes it impossible to tell which application a trace belongs to in your observability backend.

The Splunk Java agent operates at the JVM level: it starts once and applies a single configuration to every deployed application. There is no mechanism to give each deployment its own identity.

How it is solved

WildFly 27+ ships a built-in OpenTelemetry subsystem (org.wildfly.extension.opentelemetry) that creates an isolated OTel SDK instance per deployment. Each WAR reads its own OTel configuration from MicroProfile Config, so every application can declare its own service.name — and any other resource attributes — independently.

The Splunk Java agent is removed entirely. Its responsibilities are redistributed:

Responsibility	Replacement
Traces	WildFly OpenTelemetry subsystem (per-deployment, via gRPC to OTel Collector)
JVM metrics	OTel Collector Prometheus receiver scraping WildFly's built-in /metrics endpoint
Server-Timing response headers	Per-WAR JAX-RS ContainerResponseFilter

The OTel Collector is unaffected — it continues to receive standard OTLP data.

Repository structure

• start_agent.sh — start the collector, WildFly, or send a test span
• agent_config_debug.yaml — OTel Collector config (debug/local)
• java-application-server/
  • build.sh — build both WARs and copy to WildFly deployments
  • README.md — step-by-step implementation guide
  • app1/ — example JAX-RS app (calls app2)
  • app2/ — example JAX-RS app (simulated calculation, @WithSpan demo)
  • wildfly-39.0.1.Final/
    • standalone/configuration/standalone.xml — OTel subsystem config (Step 1)
    • bin/standalone.conf — JVM options (Step 2)

Prerequisites

Component	Version	Install
WildFly	39.0.1.Final	Included in java-application-server/
Java (OpenJDK)	11+	brew install openjdk
Maven	3.x	brew install maven
OTel Collector	any	Download binary or use Splunk distribution

After installing OpenJDK via Homebrew, add it to your PATH:

export PATH="/opt/homebrew/opt/openjdk/bin:$PATH"

Building the applications

./java-application-server/build.sh

This compiles both WARs and copies them into WildFly's standalone/deployments/ directory. WildFly hot-deploys them automatically if it is already running. See java-application-server/build.sh.

Starting components

Copy the example environment file once and adjust it for your local setup:

cp .env.example .env

The local .env file is loaded by start_agent.sh and is ignored by git, so it is the right place for local paths and Splunk access tokens.

Use start_agent.sh to start each component. Run it without arguments for help:

./start_agent.sh

Start the OTel Collector first, then WildFly:

./start_agent.sh collector   # starts otelcol with agent_config_debug.yaml
./start_agent.sh wildfly     # starts WildFly standalone server

The collector uses agent_config_debug.yaml.

To send a test span to verify the collector is receiving data:

./start_agent.sh testspan

Verifying it works

Once both components are running and the applications are deployed, make a request:

curl -v http://localhost:8080/app1/otherservice

The response will include a Server-Timing header with the trace context:

Server-Timing: traceparent;desc="00-<traceId>-<spanId>-01"

In the collector output you will see two separate ResourceSpans, each with a distinct service.name:

Resource attributes:
    -> service.name: Str(app1)

Resource attributes:
    -> service.name: Str(app2)

JBoss EAP

Although this project uses WildFly, the same technique applies to JBoss EAP 8.0+, which is Red Hat's supported downstream distribution of WildFly. The extension module name and MicroProfile Config mechanism are identical; the only practical difference is which subsystem XML namespace version to use. See the implementation guide for details.

Further reading

Full implementation details — including all configuration changes, the reasoning behind each decision, troubleshooting notes, and a manual instrumentation example using @WithSpan — are in java-application-server/README.md.