feat(datadog): Request secrets update from DDAgent

[atanzu]

DRAFT: DO NOT MERGE

Summary

Request secrets update from Core Agent if secrets are enabled and transactions fail with 403s.

When we use secrets, keys might become stale because of a rotation, so we want to retry the transactions later with fresh keys. To not to wait too long (and to avoid bloating of the retry queue) we add a new RPC call to request secrets update from our Core Agent.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

Add tests to cover the new functionality.

E2E test with complementary changes in Datadog Agents were not done yet.

References

• AGTMETRICS-504
• additional_endpoints API keys not refreshed dynamically at runtime #1540

This commit is complement to DataDog/datadog-agent#51480.

[datadog-datadog-prod-us1-2]

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

DataDog/saluki | check-docs     

🔧 Fix in code (Fix with Cursor).

2 errors due to spelling mistakes: 'retryable' misspelled in config.rs and mod.rs.

DataDog/saluki | check-protos     

🔧 Fix in code (Fix with Cursor).

Protocol Buffers definitions not up-to-date. Run to check out the Datadog Agent repository and commit changes.

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7a5a764 | Docs | Datadog PR Page | Give us feedback!}

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 22dc41f · Comparison: 7a5a764 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.93 MiB (baseline) vs 37.94 MiB (comparison)
Size Change: +10.09 KiB (+0.03%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
core	+75.47 KiB	1921
figment	-72.19 KiB	179
tokio	+26.85 KiB	691
resource_accounting::groups::Tracked	+24.61 KiB	7
tracing	-19.86 KiB	21
saluki_components::sources::otlp	-15.50 KiB	16
datadog_protos::trace_include::stats	-10.86 KiB	7
&mut rmp_serde	-9.22 KiB	8
axum	-8.78 KiB	46
otlp_protos::otlp_include::opentelemetry	+7.69 KiB	109
alloc	-7.31 KiB	120
serde	+7.24 KiB	11
hyper_timeout	+7.17 KiB	5
http_body_util	+6.76 KiB	62
serde_core	+6.73 KiB	93
agent_data_plane::components::ottl_filter_processor	+6.38 KiB	14
prost	-5.68 KiB	107
futures_util	-5.64 KiB	8
agent_data_plane::internal::env	-5.50 KiB	17
saluki_env::workload::stores	+4.74 KiB	6

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +68.2Ki  [NEW] +68.0Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h5c784fc005a241f7
  [NEW] +40.6Ki  [NEW] +40.5Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h88dcb18425ece695
  +0.4% +37.3Ki  +0.5% +37.9Ki    [8828 Others]
  +229% +17.0Ki  +235% +17.0Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h96c2e3d480259a38
 +13e3% +16.3Ki +33e3% +16.3Ki    core::ops::function::FnOnce::call_once::h5986ed22d7361f57
  [NEW] +12.6Ki  [NEW] +12.5Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h3100e42a5bb29be8
  [NEW] +10.0Ki  [NEW] +9.90Ki    _<resource_accounting::groups::Tracked<Inner> as core::future::future::Future>::poll::hd564e223503a4229
  +179% +9.06Ki  +184% +9.06Ki    saluki_components::transforms::trace_obfuscation::sql::obfuscate_sql_string::h9a1416d38d85025c
  [NEW] +6.99Ki  [NEW] +6.85Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h555a1a89685a9da0
  [NEW] +5.79Ki  [NEW] +5.68Ki    tokio::runtime::scheduler::current_thread::CurrentThread::block_on::h1014ea45b2d10b77
  [DEL] -6.03Ki  [DEL] -5.88Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h7bb20c45e3ab708a
  [DEL] -6.86Ki  [DEL] -6.71Ki    _<resource_accounting::groups::Tracked<Inner> as core::future::future::Future>::poll::h261c968bb4bb4caa
 -33.8% -7.07Ki -34.2% -7.07Ki    _<saluki_components::transforms::trace_obfuscation::TraceObfuscation as saluki_core::components::transforms::SynchronousTransform>::transform_buffer::h02d1d5db948ad81a
  [DEL] -11.8Ki  [DEL] -11.7Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::hdc372511b8a262d3
  [DEL] -12.3Ki  [DEL] -12.1Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hbb2339911e4ec5af
 -72.7% -12.6Ki -73.4% -12.6Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::h8ac5275f7c8ea07a
  [DEL] -15.3Ki  [DEL] -15.2Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::h0dc031a7645e6400
  [DEL] -16.1Ki  [DEL] -15.9Ki    saluki_components::sources::otlp::metrics::runtime_metrics::RUNTIME_METRICS_MAPPINGS::_{{closure}}::h1cb4c8ae104644b1
  [DEL] -17.5Ki  [DEL] -17.4Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::h223a4142ecd8db7e
  [DEL] -41.0Ki  [DEL] -40.9Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h4c31dc59c83ba86c
  [DEL] -67.3Ki  [DEL] -67.1Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h683b050b7ce873b5
  +0.0% +10.1Ki  +0.0% +11.2Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: a6ac4bbf-c00b-4090-9476-184dc6f04968
Baseline: 22dc41fa · Comparison: 7a5a764c · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
dsd_uds_1mb_3k_contexts_cpu (erratic)	cpu	⚪ +4.39	metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored)	memory	⚪ +2.04	metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic)	cpu	⚪ +1.70	metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic)	cpu	⚪ +1.57	metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic)	cpu	⚪ +1.46	metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored)	cpu	⚪ +0.76	metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput	throughput	⚪ -0.70	metrics profiles logs
otlp_ingest_metrics_5mb_memory	memory	⚪ +0.64	metrics profiles logs
dsd_uds_1mb_3k_contexts_memory	memory	⚪ +0.52	metrics profiles logs
otlp_ingest_traces_5mb_memory	memory	⚪ +0.50	metrics profiles logs
dsd_uds_10mb_3k_contexts_memory	memory	⚪ +0.34	metrics profiles logs
dsd_uds_100mb_3k_contexts_memory	memory	⚪ +0.26	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.24	metrics profiles logs
otlp_ingest_traces_5mb_throughput	throughput	⚪ -0.17	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput	throughput	⚪ -0.03	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic)	cpu	⚪ +0.03	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.03	metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput	throughput	⚪ -0.02	metrics profiles logs
otlp_ingest_metrics_5mb_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput	throughput	⚪ +0.00	metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput	throughput	⚪ +0.00	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput	throughput	⚪ +0.01	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.01	metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored)	throughput	⚪ +0.01	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory	memory	⚪ -0.04	metrics profiles logs
dsd_uds_512kb_3k_contexts_memory	memory	⚪ -0.09	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory	memory	⚪ -0.13	metrics profiles logs
dsd_uds_500mb_3k_contexts_memory	memory	⚪ -0.15	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.39	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.41	metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic)	cpu	⚪ -1.14	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic)	cpu	⚪ -1.28	metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic)	cpu	⚪ -2.89	metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic)	cpu	⚪ -2.93	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 122 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 39.7 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 59.9 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 179 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 26.7 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.