docs(agent-data-plane): Antithesis research scratchbook and bug ledger

[blt]

Summary

The Antithesis research artifacts for agent-data-plane, under test/antithesis/scratchbook/. This is
the analysis behind the harness and the bug repros:

• A SUT analysis of the DogStatsD data path and runtime.
• A property catalog (35 properties) with per-property evidence files.
• A deployment topology, a portfolio evaluation, property relationships, and a bug ledger that maps
  each discovered defect to how it is reproduced.

Docs only, no code. The internal design-partner codename is scrubbed and an internal Antithesis run
id is redacted; Confluence links and Jira references are kept.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

Docs only. check-docs is unaffected (it builds the docs/ site; these notes live under test/).
No code paths change.

References

• Builds on the harness PR (test/antithesis/) in this stack.
• The failing bug repros it catalogs are in the bug-tests PR in this stack.
• Internal context (kept per repo norms) is in the Confluence/Jira links in each artifact's
  frontmatter.

[blt]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• chore(agent-data-plane): failing repros for six discovered bugs #1769
• docs(agent-data-plane): Antithesis research scratchbook and bug ledger #1768 👈 (View in Graphite)
• chore(antithesis): Antithesis test harness and workload #1767
• enhancement(correctness): Create test/antithesis sub-project  #1774
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[datadog-prod-us1-4]

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/saluki | check-licenses     

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration.

Current 'LICENSE-3rdparty.csv' is not up to date. Please update it and commit the changes.

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3952ec9 | Docs | Datadog PR Page | Give us feedback!}

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 1bd1613 · Comparison: 3952ec9 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.89 MiB (baseline) vs 37.89 MiB (comparison)
Size Change: -328 B (-0.00%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
anon.36eead261da73468b10600f578de22ed.87.llvm.14046994766694184360	+2.83 KiB	1
anon.0acc57ca412442189c7678857beeb25f.1.llvm.18230564023335910389	-2.83 KiB	1
anon.eba192a60e6caa742f85cd2f7b63cef6.18.llvm.10590908203511297131	-2.46 KiB	1
anon.e5f0d67f8d0d8b1c6e3888577e274890.511.llvm.6538778123177323897	+2.46 KiB	1
anon.423de405463045f5e6382613e58502c9.555.llvm.7227059468892596563	+1.59 KiB	1
anon.ed85dad8c2fdcfe5c906411ebc1601c6.1.llvm.11441704272744907195	-1.59 KiB	1
anon.83861495afd7ed2d7a5f7a2024dbca33.32.llvm.16744458082958744374	+1.06 KiB	1
anon.423de405463045f5e6382613e58502c9.623.llvm.2135413985037036474	-1.06 KiB	1
anon.0cbe5d1da6c19cbc1bc3d4c0ab437541.44.llvm.197401638548642842	+822 B	1
anon.f002492f45681213d5f2178507d8f8df.914.llvm.4043333310207047691	-821 B	1
anon.d7ee623b5fafe6b057a7a0c622afa7bc.39.llvm.10556945865972081602	+652 B	1
anon.4b222af43742daee88c77b7e3b58dba9.488.llvm.10432521140313545618	-652 B	1
anon.6af111390f9ec0d8aca610283c4bc8fb.148.llvm.12481652543943855889	-652 B	1
anon.6af111390f9ec0d8aca610283c4bc8fb.148.llvm.7772407472488747350	+651 B	1
anon.281a90e99c1fd604a433b96c14d92f5a.302.llvm.7830088799502066918	+629 B	1
anon.4b222af43742daee88c77b7e3b58dba9.367.llvm.10432521140313545618	-629 B	1
_RNvXsZ_NtNtCs7BliBxkEZsN_4core3fmt3numjNtB7_5Debug3fmt.llvm.17068787117399038695	+579 B	1
_RNvXsX_NtNtCs7BliBxkEZsN_4core3fmt3numyNtB7_5Debug3fmt.llvm.12447776353896946495	-579 B	1
_RNvXsZ_NtNtCs7BliBxkEZsN_4core3fmt3numjNtB7_5Debug3fmt.llvm.10432521140313545618	-579 B	1
_RNvXsX_NtNtCs7BliBxkEZsN_4core3fmt3numyNtB7_5Debug3fmt.llvm.4062887792818234319	+578 B	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +3.43Ki  [NEW]    +456    core::ptr::drop_in_place<core::iter::adapters::map::Map<std::collections::hash::map::IntoIter<axum::routing::RouteId,axum::routing::Endpoint<saluki_components::destinations::dsd_stats::DogStatsDAPIHandlerState>>,axum::routing::path_router::PathRouter<saluki_components::destinations::dsd_stats::DogStatsDAPIHandlerState,_>::with_state<$LP$$RP$>::{{closure}}>>::h5a147bb683008716
  [NEW] +2.84Ki  [NEW]     +47    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::size_hint::h674725303a3ab093
  [NEW] +2.83Ki  [NEW]      +2    anon.36eead261da73468b10600f578de22ed.87.llvm.14046994766694184360
  [NEW] +2.46Ki  [NEW]     +80    anon.e5f0d67f8d0d8b1c6e3888577e274890.511.llvm.6538778123177323897
  [NEW] +1.59Ki  [NEW]     +97    anon.423de405463045f5e6382613e58502c9.555.llvm.7227059468892596563
  [NEW] +1.57Ki  [NEW]    +703    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::hb2bbfae5f62c1b09
  [NEW] +1.38Ki  [NEW]    +691    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::h147ce3747417d704
  [NEW] +1.29Ki  [NEW]    +315    core::ptr::drop_in_place<tokio::sync::mpsc::bounded::Permit<saluki_core::topology::interconnect::event_buffer::FixedSizeEventBuffer<1024_usize>>>::ha0685aee77bc2b0c
  +266% +1.28Ki  +100%    +168    core::ptr::drop_in_place<alloc::boxed::convert::<impl core::convert::From<alloc::string::String> for alloc::boxed::Box<dyn core::error::Error+core::marker::Send+core::marker::Sync>>::from::StringError>::h2efcd91a58f342f0
  [NEW] +1.27Ki  [NEW]     +33    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<tonic::codec::encode::EncodeBody<tonic_prost::codec::ProstEncoder<datadog_protos::agent_include::datadog::model::v1::HostTagRequest>,tokio_stream::stream_ext::map::Map<tokio_stream::once::Once<datadog_protos::agent_include::datadog::model::v1::HostTagRequest>,core::result::Result<datadog_protos::agent_include::datadog::model::v1::HostTagRequest,tonic::status::Status>::Ok>>,tonic::status::Status::map_error<tonic::status::Status>>>::h12a2caa920ba5408
  -0.1%    -325  [ = ]       0    [1566 Others]
  [DEL] -1.27Ki  [DEL]     -33    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<tonic::codec::encode::EncodeBody<tonic_prost::codec::ProstEncoder<datadog_protos::agent_include::datadog::model::v1::HostnameRequest>,tokio_stream::stream_ext::map::Map<tokio_stream::once::Once<datadog_protos::agent_include::datadog::model::v1::HostnameRequest>,core::result::Result<datadog_protos::agent_include::datadog::model::v1::HostnameRequest,tonic::status::Status>::Ok>>,tonic::status::Status::map_error<tonic::status::Status>>>::h7d0bcb55600c1599
 -40.4% -1.28Ki -14.3%    -168    core::ptr::drop_in_place<alloc::string::String>::hf4631bd4382d6119
  [DEL] -1.29Ki  [DEL]    -315    core::ptr::drop_in_place<tokio::sync::mpsc::bounded::Permit<saluki_env::workload::metadata::MetadataOperation>>::h2e1e385cf4a89858
  [DEL] -1.38Ki  [DEL]    -691    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::h8d3d490742ef5359
  [DEL] -1.57Ki  [DEL]    -703    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::h4c56918971715b8b
  [DEL] -1.59Ki  [DEL]     -97    anon.ed85dad8c2fdcfe5c906411ebc1601c6.1.llvm.11441704272744907195
  [DEL] -2.46Ki  [DEL]     -80    anon.eba192a60e6caa742f85cd2f7b63cef6.18.llvm.10590908203511297131
  [DEL] -2.83Ki  [DEL]      -2    anon.0acc57ca412442189c7678857beeb25f.1.llvm.18230564023335910389
  [DEL] -2.84Ki  [DEL]     -47    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::size_hint::h66dcc7212f74260d
  [DEL] -3.44Ki  [DEL]    -456    core::ptr::drop_in_place<core::iter::adapters::map::Map<std::collections::hash::map::IntoIter<axum::routing::RouteId,axum::routing::Endpoint<saluki_components::sources::dogstatsd::replay::replay_control::DogStatsDReplayControl>>,axum::routing::path_router::PathRouter<saluki_components::sources::dogstatsd::replay::replay_control::DogStatsDReplayControl,_>::with_state<$LP$$RP$>::{{closure}}>>::h9e288300660bcc12
  -0.0%    -328  [ = ]       0    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 7fbeff23-16b0-4f7b-a73c-8f0ca997e681
Baseline: 1bd16137 · Comparison: 3952ec9c · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
dsd_uds_1mb_3k_contexts_cpu (erratic)	cpu	⚪ +12.00	metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic)	cpu	⚪ +6.89	metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored)	memory	⚪ +2.47	metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic)	cpu	⚪ +1.24	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.32	metrics profiles logs
otlp_ingest_metrics_5mb_memory	memory	⚪ +0.28	metrics profiles logs
otlp_ingest_traces_5mb_memory	memory	⚪ +0.25	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic)	cpu	⚪ +0.12	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.11	metrics profiles logs
otlp_ingest_traces_5mb_throughput	throughput	⚪ -0.11	metrics profiles logs
dsd_uds_100mb_3k_contexts_memory	memory	⚪ +0.09	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput	throughput	⚪ -0.06	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.05	metrics profiles logs
dsd_uds_1mb_3k_contexts_memory	memory	⚪ +0.03	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory	memory	⚪ +0.02	metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput	throughput	⚪ -0.01	metrics profiles logs
otlp_ingest_metrics_5mb_throughput	throughput	⚪ -0.01	metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput	throughput	⚪ +0.00	metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored)	throughput	⚪ +0.02	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput	throughput	⚪ +0.04	metrics profiles logs
dsd_uds_500mb_3k_contexts_memory	memory	⚪ -0.04	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic)	cpu	⚪ -0.05	metrics profiles logs
dsd_uds_512kb_3k_contexts_memory	memory	⚪ -0.09	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.16	metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic)	cpu	⚪ -0.26	metrics profiles logs
dsd_uds_10mb_3k_contexts_memory	memory	⚪ -0.30	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.32	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory	memory	⚪ -0.35	metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic)	cpu	⚪ -0.52	metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic)	cpu	⚪ -0.71	metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput	throughput	⚪ +1.08	metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored)	cpu	⚪ -1.16	metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic)	cpu	⚪ -2.74	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 126 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 39.8 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 60.4 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 181 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 26.6 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc4bb29728

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Remove stale aggregate panic repro

This ledger is being added on top of parent 476a5b6, where AggregateConfiguration already deserializes aggregate_window_duration_seconds as NonZeroU64 and align_to_bucket_start takes a NonZeroU64, so a zero bucket width cannot reach the modulo path; zero or fractional-second values fail configuration parsing instead. Keeping this as a current failing repro sends the Antithesis bug work toward an already-fixed/non-reproducible defect and also invalidates the “expect six FAILURES” command below.

Useful? React with 👍 / 👎.

[Copilot]

Pull request overview

Adds Antithesis research documentation for the agent-data-plane DogStatsD scope under test/antithesis/scratchbook/, including property evidence, evaluation synthesis, and a bug ledger.

Changes:

• Adds per-property evidence files for memory, aggregation, config, lifecycle, retry, replay, parsing, and event/service-check behavior.
• Adds portfolio-level relationship and synthesis documents.
• Adds a bug ledger mapping discovered defects to repro vehicles or follow-up Antithesis shots.

Reviewed changes

Copilot reviewed 46 out of 46 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/antithesis/scratchbook/property-relationships.md	Groups properties into shared mechanism clusters.
test/antithesis/scratchbook/existing-assertions.md	Documents absence of Antithesis SDK assertions.
test/antithesis/scratchbook/evaluation/synthesis.md	Summarizes evaluation lenses, added gaps, refinements, and scope decisions.
test/antithesis/scratchbook/bug-ledger.md	Tracks discovered defects and repro/triage status.
test/antithesis/scratchbook/properties/*.md	Adds detailed evidence and proposed assertions for individual properties across ADP behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.