[OpAMP] Snapshot profiling preparation for remote config by robsunday · Pull Request #2944 · signalfx/splunk-otel-java

robsunday · 2026-07-15T06:55:37Z

This is refactoring of snapshot profiler preparing it to dynamic enabling and disabling with remote config.
While PR touches a lot of files, most of them are unit tests.

Snapshot profiling was implemented using multiple components that were mostly initialized during SDK customization.
These components are:

StackTraceSampler
StagingArea
StackTraceExporter
ActiveSpanTracker
TraceThreadChangeDetector
SnapshotProfilingSpanProcessor

With this PR the following changes was introduced to enable snapshot profiling after agent started:

Snapshot profiling is initiated after SDK is initialized, from SnapshotProfilingAgentListener.afterAgent.
StackTraceSampler, StagingArea and StackTraceExporter are created when snapshot profiling is enabled.
ActiveSpanTracker, TraceThreadChangeDetector and SnapshotProfilingSpanProcessor are always created but they are disabled by default. They are enabled when snapshot profiling is enabled. When in disabled state, they add close to zero overhead.

This PR does not implement support for remote config. It is necessary preparation and dynamic enable/disable will be delivered by subsequent PRs.

robsunday · 2026-07-17T12:45:47Z


-    if (snapshotProfiling.isEnabled()) {
-      initActiveSpansTracking();
-      initStackTraceSampler(snapshotProfiling);


[for reviewer] StackTraceSampler is now initiated in StackTraceSamplerInitializer called by SnapshotProfilingSupervisor

robsunday · 2026-07-17T12:48:09Z

  @Override
  public Scope attach(Context toAttach) {
    Scope scope = delegate.attach(toAttach);
+    if (!isEnabled()) {


[for reviewer] ActiveSpanTracker is always created during SDK initialization and can be turned on and off by SnapshotProfilingSupervisor

This is the unfortunate world we will live in forever now. 💣

robsunday · 2026-07-17T12:50:14Z

-      if (snapshotProfilingEnabled()) {
-        builder.addSpanProcessor(new SdkShutdownHook());
-      }
+      builder.addSpanProcessor(new SdkShutdownHook());


[for reviewer] It is safe to call SdkShutdownHook regardless if snapshot profiling is on or off

robsunday · 2026-07-17T12:53:11Z

[for reviewer] Like in SnapshotProfilingConfigurationCustomizerProvider, we always preconfigure some components that later can be enabled or disabled by SnapshotProfilingSupervisor

robsunday · 2026-07-17T12:55:01Z

    SnapshotProfilingConfiguration snapshotProfiling = getSnapshotProfilingConfig(model);
    SnapshotProfilingConfiguration.SUPPLIER.configure(snapshotProfiling);

-    if (snapshotProfiling.isEnabled()) {


[for reviewer] We always setup some components that must be created during sdk initialization, and later on they are enabled or disabled by SnapshotProfilingSupervisor.

robsunday · 2026-07-21T16:25:56Z

  private Function<ConfigProperties, Map<String, String>> startTrackingActiveSpans(
      TraceRegistry registry) {
    return properties -> {
-      if (snapshotProfilingEnabled()) {


[for reviewer] This check is replaced by enabled property in ActiveSpanTracker and TraceThreadChangeDetector that are created by ContextStorageWrapper

robsunday · 2026-07-21T16:28:25Z

-  private Function<ConfigProperties, Map<String, String>> setupStackTraceSampler() {
-    return properties -> {
-      if (snapshotProfilingEnabled()) {
-        StackTraceSampler sampler = samplerProvider.apply(properties);


[for reviewer] It is now created by StackTraceSamplerInitializer called by SnapshotProfilingSupervisor when profiling is enabled

robsunday · 2026-07-21T16:44:05Z


    autoConfigurationCustomizer
        .addTracerProviderCustomizer(snapshotProfilingSpanProcessor(registry))
-        .addPropertiesCustomizer(setupStackTraceSampler())


[for reviewer] StackTraceSampler is now initiated in StackTraceSamplerInitializer called by SnapshotProfilingSupervisor

robsunday · 2026-07-21T16:45:15Z

-  private boolean includeTraceContextPropagator(Set<String> configuredPropagators) {
-    return configuredPropagators.isEmpty();
-  }
+      SnapshotProfilingSpanProcessor spanProcessor =


[for reviewer] SnapshotProfilingSpanProcessor is now enabled and disabled by SnapshotProfilingSupervisor

robsunday · 2026-07-21T16:48:26Z

+
+    configurationSupplier.get().log();
+
+    StackTraceSamplerInitializer.setupStackTraceSampler(configurationSupplier.get());


[for reviewer] I'll probably move these methods to SnapshotProfilingSupervisor class

breedx-splk · 2026-07-23T17:58:52Z

      return;
    }
+
+    updateJvmMemoryMetrics();


wanna move it down below the jfr check too then?

I was not sure. It can be moved if we are sure we do not want to gather these metrics in case JFR is not available.
Do you recommend to move it down?

Anyway, I moved it down. In case we decide to keep it upper I can do revert it in next PR. If these metrics do not make sense without profiling data then it should be fine now.

breedx-splk · 2026-07-23T18:24:56Z

+  static void setupStackTraceExporter(
+      SnapshotProfilingConfiguration configuration,
+      Resource resource,
+      OtelLoggerFactory otelLoggerFactory) {
+    int maxDepth = configuration.getStackDepth();
+    Logger otelLogger =
+        buildLogger(otelLoggerFactory, resource, configuration.getConfigProperties());
+    AsyncStackTraceExporter exporter = new AsyncStackTraceExporter(otelLogger, maxDepth);
+    StackTraceExporter.SUPPLIER.configure(exporter);
+  }
+
  private static StagingArea createStagingArea(SnapshotProfilingConfiguration configuration) {
    Duration interval = configuration.getExportInterval();
    int capacity = configuration.getStagingCapacity();
    return new PeriodicallyExportingStagingArea(StackTraceExporter.SUPPLIER, interval, capacity);
  }
+
+  private static Logger buildLogger(
+      OtelLoggerFactory otelLoggerFactory, Resource resource, Object configProperties) {
+    if (configProperties instanceof DeclarativeConfigProperties) {
+      DeclarativeConfigProperties exporterConfig =
+          DeclarativeConfigPropertiesUtil.getStructuredOrEmpty(
+              (DeclarativeConfigProperties) configProperties, "exporter");
+      return otelLoggerFactory.build(exporterConfig, resource);
+    }
+    if (configProperties instanceof ConfigProperties) {
+      return otelLoggerFactory.build((ConfigProperties) configProperties, resource);
+    }
+    throw new IllegalArgumentException(
+        "Unsupported config properties type: " + configProperties.getClass().getName());
+  }


It's a little weird to me now that a class whose name suggests that it initializes a stack trace sampler, now also builds a logger and sets up an exporter. Should that be a separate responsibility?

Agreed. This class no longer serves its original purpose. I already refactored it in #2955 (moved to SnapshotProfilingSupervisor), but I think I can improve it even further.

breedx-splk · 2026-07-23T18:31:33Z

  private final TraceRegistry registry;
  private final Supplier<StackTraceSampler> sampler;

+  private boolean enabled;


Should this be volatile?

breedx-splk

Thanks for getting this started. I had a few small comments, but I think it's fine to proceed with this.

I think the tests here do a pretty good job of showing how much unfortunate static/global state we are introducing, especially with all of the SUPPLIER stuff everywhere, but that's just what we got now I guess.

I appreciate that it's mostly following the same patterns as the main profiler mess, which made it easier to review...but I was a little surprised that the supervisor wasn't "command" based. Maybe that is happening in a subsequent PR? Anyway, cheers.

robsunday · 2026-07-24T18:50:26Z

but I was a little surprised that the supervisor wasn't "command" based. Maybe that is happening in a subsequent PR?

Yes. I have a bit different idea which I'm going to discuss with you first.

robsunday added 9 commits July 7, 2026 12:21

Snapshot Profiler Configuration refactoring

bf44697

Fix for missing nodes in profiler exporter config

5c93e2b

Fix for missing nodes in profiler exporter config

efcacf6

Code review followup

eff6544

Merge branch 'main' into opamp-snapshot-profiler-config-refactor

b8e36cf

Merge branch 'main' into opamp-snapshot-profiler-config-refactor

57276b0

Initial commit

898d25e

Test fixes and cleanup of comments

73b5c6d

Merge branch 'main' into opamp-call-graphs

dbdaeaa

robsunday changed the title ~~Opamp call graphs~~ [OpAMP] Enabling and disabling call graphs via remote config Jul 15, 2026

robsunday added 7 commits July 15, 2026 09:17

Tests cleanup

576149f

SpanTracker support

6f66ce4

Merge branch 'main' into opamp-call-graphs

cce4a7d

TraceThreadChangeDetector support

656daea

Tests updated after TraceThreadChangeDetector support

900e1f8

Tests improved

a7aa8c6

Declarative config customizer improvements

0538bb5

robsunday commented Jul 17, 2026

View reviewed changes

robsunday added 3 commits July 20, 2026 16:08

SnapshotProfilingSpanProcessor support

b23d3d1

spotless

e39a53f

Merge branch 'main' into opamp-call-graphs

a1c515e

robsunday marked this pull request as ready for review July 21, 2026 11:17

robsunday requested review from a team as code owners July 21, 2026 11:17

robsunday changed the title ~~[OpAMP] Enabling and disabling call graphs via remote config~~ [OpAMP] Snapshot profiling preparation for remote config Jul 21, 2026

robsunday commented Jul 21, 2026

View reviewed changes

robsunday mentioned this pull request Jul 21, 2026

[OpAMP] Enable/disable snapshot profiling with remote config #2955

Open

Merge branch 'main' into opamp-call-graphs

d8bd09c

breedx-splk reviewed Jul 23, 2026

View reviewed changes

breedx-splk approved these changes Jul 23, 2026

View reviewed changes

robsunday added 4 commits July 24, 2026 09:56

code review followup

c084413

Merge branch 'main' into opamp-call-graphs

c603258

code review followup

4ac828d

Merge branch 'main' into opamp-call-graphs

0cac85b

robsunday merged commit ba88e5d into signalfx:main Jul 24, 2026
28 checks passed

github-actions Bot locked and limited conversation to collaborators Jul 24, 2026


		configurationSupplier.get().log();

		StackTraceSamplerInitializer.setupStackTraceSampler(configurationSupplier.get());

Uh oh!

Conversation

robsunday commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robsunday Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robsunday Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robsunday Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

breedx-splk left a comment

Choose a reason for hiding this comment

Uh oh!

robsunday commented Jul 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

robsunday commented Jul 15, 2026 •

edited

Loading

robsunday Jul 17, 2026 •

edited

Loading

robsunday Jul 21, 2026 •

edited

Loading

robsunday Jul 24, 2026 •

edited

Loading