Report agentic onboarding marker in configuration telemetry (RFC-1113) by christophe-papazian · Pull Request #6079 · DataDog/dd-trace-rb

christophe-papazian · 2026-07-22T08:58:29Z

APPSEC-69231

What does this PR do?
Registers DD_APPSEC_AGENTIC_ONBOARDING as an ordinary string configuration key (default empty), reported verbatim in the app-started configuration telemetry through the standard config pipeline. It is always emitted — empty value with origin=default when unset. No derived boolean and no read of AppSec's runtime-enabled state.

Motivation:
RFC-1113: a runtime telemetry signal that a service was onboarded by the AAP agentic onboarding solution, so the usage-to-activation funnel can be measured in production. Ports DataDog/dd-trace-java#11983, DataDog/dd-trace-dotnet#8920, DataDog/dd-trace-go#5055.

Change log entry
None.

How to test the change?
Covered by unit specs: spec/datadog/appsec/configuration/settings_spec.rb (unset → "", env value passed through verbatim) and spec/datadog/core/telemetry/event/app_started_spec.rb (always reported with empty value and origin=default when unset).

Register DD_APPSEC_AGENTIC_ONBOARDING as an ordinary string configuration key, reported verbatim in app-started configuration telemetry. Always emitted: empty value with origin=default when unset. No derived boolean and no read of AppSec's runtime-enabled state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

datadog-prod-us1-5 · 2026-07-22T09:01:54Z

Tests

⚠️ Warnings

🚦 15 Pipeline jobs failed

Unit Tests | Ruby 2.5 / build & test (standard) [0]

Unit Tests | Ruby 3.0 / build & test (standard) [0]

Unit Tests | Ruby 3.1 / build & test (standard) [0]

View all 15 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 252ac05 | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-07-22T09:27:48Z

Benchmarks

Benchmark execution time: 2026-07-23 07:55:55

Comparing candidate commit 252ac05 in PR branch christophe-papazian/appsec-agentic-onboarding-telemetry with baseline commit 3d5a34c in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:tracing - trace.to_digest - Continue

unstable throughput [-1548.439op/s; +1541.061op/s] or [-5.270%; +5.245%]

The previously pinned ref predated the RFC-1113 rewrite of test_agentic_onboarding.py (it still had the RFC-1110 derived-boolean test). Bump to system-tests main so the agentic-onboarding config telemetry tests run against this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ing-telemetry

…tests" This reverts commit b9ea772.

p-datadog · 2026-07-22T13:53:07Z

+              option :agentic_onboarding do |o|
+                o.type :string
+                o.env "DD_APPSEC_AGENTIC_ONBOARDING"
+                o.default ""


What was the reasoning for using an empty string instead of nil? Or false if the type is indeed supposed to be bool?

null is not always reported, empty string is.
This is based on an RFC that was discussed for some time, if you want the full context, we can plan a meeting.

p-datadog · 2026-07-22T13:54:38Z

              end

+              # Set by the agentic onboarding solution; no behavior, reported verbatim in
+              # configuration telemetry only (RFC-1113).


This looks like AI slop. The comment (or the rest of PR) does not provide a description of what the value actually contains.

Some people wants no comments, you seem to want all the explanation in here ? The Appsec RFC ID is put here in reference, so anyone can take a look at it for more explanations.

RFC ID is put here in reference

Please remove the RFC reference and JIRA numbers from code comment and pull request description and title.

Replace with an example how agentic onboarding solution is going to modify this specific settings. Provided with possible values.

p-datadog · 2026-07-22T13:54:54Z

+              # Set by the agentic onboarding solution; no behavior, reported verbatim in
+              # configuration telemetry only (RFC-1113).
+              option :agentic_onboarding do |o|
+                o.type :string


The tests set the value to a boolean, which means the type should probably be bool and not string.

It is defined as a string, both in the RFC and the registry

Please also note that the CI shouldn't pass if the configuration was not defined as the registry is.

…ing-telemetry

p-datadog · 2026-07-24T17:12:41Z

+        end
+
+        context "is not defined" do
+          let(:agentic_onboarding_env) { nil }


I commented on the RFC. This test case converts the "not present" case (env var is nil) to "present" case (env var is set to empty string). The RFC specifies that the value of the env var is ignored, therefore the test missing from the PR is env var is set to empty string -> reported value is true.

p-datadog · 2026-07-24T17:13:31Z

+
+      before { set_agentic_onboarding }
+
+      it { expect(settings.appsec.agentic_onboarding).to eq("true") }


The RFC specifies that telemetry signal is a boolean. Therefore the string -> bool conversion should probably be done at configuration parsing time and this test should be written to expect a boolean value.

christophe-papazian added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jul 22, 2026

dd-octo-sts Bot added core Involves Datadog core libraries appsec Application Security monitoring product labels Jul 22, 2026

christophe-papazian and others added 2 commits July 22, 2026 11:58

Merge branch 'master' into christophe-papazian/appsec-agentic-onboard…

d849cfb

…ing-telemetry

christophe-papazian marked this pull request as ready for review July 22, 2026 11:33

christophe-papazian requested review from a team as code owners July 22, 2026 11:33

Revert "Bump system-tests pin to include RFC-1113 agentic-onboarding …

d1026f2

…tests" This reverts commit b9ea772.

This was referenced Jul 22, 2026

APPSEC-69232 Report DD_APPSEC_AGENTIC_ONBOARDING in config telemetry DataDog/dd-trace-php#4053

Merged

feat(appsec): report DD_APPSEC_AGENTIC_ONBOARDING in config telemetry DataDog/dd-trace-js#9486

Merged

p-datadog reviewed Jul 22, 2026

View reviewed changes

Merge branch 'master' into christophe-papazian/appsec-agentic-onboard…

252ac05

…ing-telemetry

TonyCTHsu requested changes Jul 23, 2026

View reviewed changes

p-datadog reviewed Jul 24, 2026

View reviewed changes


		before { set_agentic_onboarding }

		it { expect(settings.appsec.agentic_onboarding).to eq("true") }

Uh oh!

Conversation

christophe-papazian commented Jul 22, 2026

Uh oh!

datadog-prod-us1-5 Bot commented Jul 22, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

pr-commenter Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

Unstable benchmarks

scenario:tracing - trace.to_digest - Continue

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TonyCTHsu Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christophe-papazian Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

datadog-prod-us1-5 Bot commented Jul 22, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

pr-commenter Bot commented Jul 22, 2026 •

edited

Loading

TonyCTHsu Jul 23, 2026 •

edited

Loading

christophe-papazian Jul 22, 2026 •

edited

Loading