Skip to content

chore(antithesis): Antithesis test harness and workload#1767

Open
blt wants to merge 1 commit into
mainfrom
blt/antithesis-harness
Open

chore(antithesis): Antithesis test harness and workload#1767
blt wants to merge 1 commit into
mainfrom
blt/antithesis-harness

Conversation

@blt
Copy link
Copy Markdown
Contributor

@blt blt commented May 29, 2026

Summary

Adds an Antithesis test harness for agent-data-plane under test/antithesis/. It brings ADP up in a minimal, runnable topology and gives Antithesis a workload to drive. Right now I've mostly yanked things from millstone. I will expand on this in follow-on PRs.

The new antithesis feature and SDK are not enabled by default, so production builds are unchanged.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

I fired off a job to our antithesis tenant cluster.

References

N/A

@dd-octo-sts dd-octo-sts Bot added the area/test All things testing: unit/integration, correctness, SMP regression, etc. label May 29, 2026
Copy link
Copy Markdown
Contributor Author

blt commented May 29, 2026

@datadog-datadog-prod-us1-2
Copy link
Copy Markdown

datadog-datadog-prod-us1-2 Bot commented May 29, 2026

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/saluki | check-licenses   View in Datadog   GitLab

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration. Current 'LICENSE-3rdparty.csv' is not up to date.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 476a5b6 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented May 29, 2026

Binary Size Analysis (Agent Data Plane)

Baseline: 1bd1613 · Comparison: 476a5b6 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.89 MiB (baseline) vs 37.89 MiB (comparison)
Size Change: -328 B (-0.00%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
anon.36eead261da73468b10600f578de22ed.87.llvm.14046994766694184360 +2.83 KiB 1
anon.0acc57ca412442189c7678857beeb25f.1.llvm.18230564023335910389 -2.83 KiB 1
anon.eba192a60e6caa742f85cd2f7b63cef6.18.llvm.10590908203511297131 -2.46 KiB 1
anon.e5f0d67f8d0d8b1c6e3888577e274890.511.llvm.6538778123177323897 +2.46 KiB 1
anon.423de405463045f5e6382613e58502c9.555.llvm.7227059468892596563 +1.59 KiB 1
anon.ed85dad8c2fdcfe5c906411ebc1601c6.1.llvm.11441704272744907195 -1.59 KiB 1
anon.83861495afd7ed2d7a5f7a2024dbca33.32.llvm.16744458082958744374 +1.06 KiB 1
anon.423de405463045f5e6382613e58502c9.623.llvm.2135413985037036474 -1.06 KiB 1
anon.0cbe5d1da6c19cbc1bc3d4c0ab437541.44.llvm.197401638548642842 +822 B 1
anon.f002492f45681213d5f2178507d8f8df.914.llvm.4043333310207047691 -821 B 1
anon.d7ee623b5fafe6b057a7a0c622afa7bc.39.llvm.10556945865972081602 +652 B 1
anon.4b222af43742daee88c77b7e3b58dba9.488.llvm.10432521140313545618 -652 B 1
anon.6af111390f9ec0d8aca610283c4bc8fb.148.llvm.12481652543943855889 -652 B 1
anon.6af111390f9ec0d8aca610283c4bc8fb.148.llvm.7772407472488747350 +651 B 1
anon.281a90e99c1fd604a433b96c14d92f5a.302.llvm.7830088799502066918 +629 B 1
anon.4b222af43742daee88c77b7e3b58dba9.367.llvm.10432521140313545618 -629 B 1
_RNvXsZ_NtNtCs7BliBxkEZsN_4core3fmt3numjNtB7_5Debug3fmt.llvm.17068787117399038695 +579 B 1
_RNvXsX_NtNtCs7BliBxkEZsN_4core3fmt3numyNtB7_5Debug3fmt.llvm.12447776353896946495 -579 B 1
_RNvXsZ_NtNtCs7BliBxkEZsN_4core3fmt3numjNtB7_5Debug3fmt.llvm.10432521140313545618 -579 B 1
_RNvXsX_NtNtCs7BliBxkEZsN_4core3fmt3numyNtB7_5Debug3fmt.llvm.4062887792818234319 +578 B 1
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +3.43Ki  [NEW]    +456    core::ptr::drop_in_place<core::iter::adapters::map::Map<std::collections::hash::map::IntoIter<axum::routing::RouteId,axum::routing::Endpoint<saluki_components::destinations::dsd_stats::DogStatsDAPIHandlerState>>,axum::routing::path_router::PathRouter<saluki_components::destinations::dsd_stats::DogStatsDAPIHandlerState,_>::with_state<$LP$$RP$>::{{closure}}>>::h5a147bb683008716
  [NEW] +2.84Ki  [NEW]     +47    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::size_hint::h674725303a3ab093
  [NEW] +2.83Ki  [NEW]      +2    anon.36eead261da73468b10600f578de22ed.87.llvm.14046994766694184360
  [NEW] +2.46Ki  [NEW]     +80    anon.e5f0d67f8d0d8b1c6e3888577e274890.511.llvm.6538778123177323897
  [NEW] +1.59Ki  [NEW]     +97    anon.423de405463045f5e6382613e58502c9.555.llvm.7227059468892596563
  [NEW] +1.57Ki  [NEW]    +703    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::hb2bbfae5f62c1b09
  [NEW] +1.38Ki  [NEW]    +691    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::h147ce3747417d704
  [NEW] +1.29Ki  [NEW]    +315    core::ptr::drop_in_place<tokio::sync::mpsc::bounded::Permit<saluki_core::topology::interconnect::event_buffer::FixedSizeEventBuffer<1024_usize>>>::ha0685aee77bc2b0c
  +266% +1.28Ki  +100%    +168    core::ptr::drop_in_place<alloc::boxed::convert::<impl core::convert::From<alloc::string::String> for alloc::boxed::Box<dyn core::error::Error+core::marker::Send+core::marker::Sync>>::from::StringError>::h2efcd91a58f342f0
  [NEW] +1.27Ki  [NEW]     +33    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<tonic::codec::encode::EncodeBody<tonic_prost::codec::ProstEncoder<datadog_protos::agent_include::datadog::model::v1::HostTagRequest>,tokio_stream::stream_ext::map::Map<tokio_stream::once::Once<datadog_protos::agent_include::datadog::model::v1::HostTagRequest>,core::result::Result<datadog_protos::agent_include::datadog::model::v1::HostTagRequest,tonic::status::Status>::Ok>>,tonic::status::Status::map_error<tonic::status::Status>>>::h12a2caa920ba5408
  -0.1%    -325  [ = ]       0    [1566 Others]
  [DEL] -1.27Ki  [DEL]     -33    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<tonic::codec::encode::EncodeBody<tonic_prost::codec::ProstEncoder<datadog_protos::agent_include::datadog::model::v1::HostnameRequest>,tokio_stream::stream_ext::map::Map<tokio_stream::once::Once<datadog_protos::agent_include::datadog::model::v1::HostnameRequest>,core::result::Result<datadog_protos::agent_include::datadog::model::v1::HostnameRequest,tonic::status::Status>::Ok>>,tonic::status::Status::map_error<tonic::status::Status>>>::h7d0bcb55600c1599
 -40.4% -1.28Ki -14.3%    -168    core::ptr::drop_in_place<alloc::string::String>::hf4631bd4382d6119
  [DEL] -1.29Ki  [DEL]    -315    core::ptr::drop_in_place<tokio::sync::mpsc::bounded::Permit<saluki_env::workload::metadata::MetadataOperation>>::h2e1e385cf4a89858
  [DEL] -1.38Ki  [DEL]    -691    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::h8d3d490742ef5359
  [DEL] -1.57Ki  [DEL]    -703    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::h4c56918971715b8b
  [DEL] -1.59Ki  [DEL]     -97    anon.ed85dad8c2fdcfe5c906411ebc1601c6.1.llvm.11441704272744907195
  [DEL] -2.46Ki  [DEL]     -80    anon.eba192a60e6caa742f85cd2f7b63cef6.18.llvm.10590908203511297131
  [DEL] -2.83Ki  [DEL]      -2    anon.0acc57ca412442189c7678857beeb25f.1.llvm.18230564023335910389
  [DEL] -2.84Ki  [DEL]     -47    _<http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::size_hint::h66dcc7212f74260d
  [DEL] -3.44Ki  [DEL]    -456    core::ptr::drop_in_place<core::iter::adapters::map::Map<std::collections::hash::map::IntoIter<axum::routing::RouteId,axum::routing::Endpoint<saluki_components::sources::dogstatsd::replay::replay_control::DogStatsDReplayControl>>,axum::routing::path_router::PathRouter<saluki_components::sources::dogstatsd::replay::replay_control::DogStatsDReplayControl,_>::with_state<$LP$$RP$>::{{closure}}>>::h9e288300660bcc12
  -0.0%    -328  [ = ]       0    TOTAL

@blt blt changed the title feat(antithesis): test harness + workload for agent-data-plane chore(agent-data-plane): Antithesis test harness and workload May 29, 2026
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented May 29, 2026

Regression Detector (Agent Data Plane)

Run ID: 0612d7ab-1abc-436f-981d-dfc1e6403cbe
Baseline: 1bd16137 · Comparison: 476a5b66 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
otlp_ingest_traces_5mb_cpu (erratic) cpu ⚪ +2.61 metrics profiles logs
otlp_ingest_metrics_5mb_memory memory ⚪ +1.82 metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic) cpu ⚪ +1.07 metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput throughput ⚪ -1.00 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic) cpu ⚪ +0.46 metrics profiles logs
dsd_uds_500mb_3k_contexts_memory memory ⚪ +0.40 metrics profiles logs
otlp_ingest_traces_5mb_memory memory ⚪ +0.19 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput throughput ⚪ -0.11 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ +0.11 metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory ⚪ +0.10 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ +0.10 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory memory ⚪ +0.09 metrics profiles logs
quality_gates_rss_idle memory ⚪ +0.08 metrics profiles logs
dsd_uds_1mb_3k_contexts_cpu (erratic) cpu ⚪ +0.07 metrics profiles logs
otlp_ingest_traces_5mb_throughput throughput ⚪ -0.04 metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored) throughput ⚪ -0.02 metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput throughput ⚪ +0.01 metrics profiles logs
otlp_ingest_metrics_5mb_throughput throughput ⚪ +0.01 metrics profiles logs
dsd_uds_10mb_3k_contexts_memory memory ⚪ -0.04 metrics profiles logs
dsd_uds_512kb_3k_contexts_memory memory ⚪ -0.11 metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored) cpu ⚪ -0.17 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory memory ⚪ -0.20 metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic) cpu ⚪ -0.26 metrics profiles logs
dsd_uds_1mb_3k_contexts_memory memory ⚪ -0.41 metrics profiles logs
dsd_uds_100mb_3k_contexts_memory memory ⚪ -0.61 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput throughput ⚪ +0.71 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ -0.79 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic) cpu ⚪ -2.67 metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic) cpu ⚪ -2.93 metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic) cpu ⚪ -3.80 metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored) memory ⚪ -8.44 metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic) cpu 🟢 -12.05 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 124 MiB ≤ 140 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 39.9 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 60.7 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 186 MiB ≤ 200 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 26.6 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

@blt blt force-pushed the blt/antithesis-harness branch 3 times, most recently from 25934ae to 330a22f Compare May 29, 2026 20:34
@blt blt changed the base branch from main to graphite-base/1767 May 29, 2026 20:38
@blt blt force-pushed the graphite-base/1767 branch from 695ecfa to 8b14d24 Compare May 29, 2026 20:38
@blt blt force-pushed the blt/antithesis-harness branch from 330a22f to f8bab09 Compare May 29, 2026 20:38
@blt blt changed the base branch from graphite-base/1767 to blt/create_tests_antithesis_sub-project May 29, 2026 20:38
@blt blt force-pushed the blt/antithesis-harness branch from f8bab09 to d048536 Compare May 29, 2026 20:46
@blt blt force-pushed the blt/create_tests_antithesis_sub-project branch from 8b14d24 to 29f9642 Compare May 29, 2026 20:46
@blt blt force-pushed the blt/antithesis-harness branch from d048536 to e4defed Compare May 29, 2026 20:55
@blt blt force-pushed the blt/create_tests_antithesis_sub-project branch from 29f9642 to 1de5717 Compare May 29, 2026 20:55
Base automatically changed from blt/create_tests_antithesis_sub-project to main May 29, 2026 20:56
@blt blt changed the title chore(agent-data-plane): Antithesis test harness and workload chore(antithesis): Antithesis test harness and workload May 29, 2026
@blt blt force-pushed the blt/antithesis-harness branch 2 times, most recently from 9994924 to e46f656 Compare May 29, 2026 22:36
@blt blt marked this pull request as ready for review May 29, 2026 22:39
@blt blt requested review from a team as code owners May 29, 2026 22:39
@blt blt marked this pull request as draft May 29, 2026 22:40
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e46f6562d6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/antithesis/crates/workload-driver/src/bin/finally_verify_delivery.rs Outdated
@blt blt force-pushed the blt/antithesis-harness branch from e46f656 to fb9e655 Compare May 30, 2026 00:43
@blt blt marked this pull request as ready for review May 30, 2026 00:43
@blt blt force-pushed the blt/antithesis-harness branch from fb9e655 to 4d533a8 Compare May 30, 2026 00:47
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fb9e655158

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .dockerignore
@blt blt force-pushed the blt/antithesis-harness branch 2 times, most recently from ac323be to 6b922f8 Compare May 30, 2026 00:52
Adds an Antithesis test harness for agent-data-plane under `test/antithesis/`. It brings ADP up in a
minimal, runnable topology and gives Antithesis a workload to drive:

- An instrumented ADP image (LLVM SanitizerCoverage), gated behind a new, off-by-default `antithesis`
  cargo feature, with DWARF symbols exposed under `/symbols`.
- A mock `datadog-intake` and a `millstone`-based workload (the `adp-antithesis-workload` crate: a
  high-cardinality DogStatsD driver plus an end-to-end delivery verifier).
- `docker-compose.yaml`, a standalone ADP config with a self-signed IPC cert, `setup_complete`
  wiring, and a `.dockerignore` re-include so `test/antithesis/` reaches the Docker build context.

The `antithesis` feature and SDK are net-new and off by default, so production builds are unchanged.

- [ ] Bug fix
- [ ] New feature
- [x] Non-functional (chore, refactoring, docs)
- [ ] Performance

- `docker compose -f test/antithesis/config/docker-compose.yaml build` builds all three images
  (amd64; the in-build instrumentation symbol checks pass).
- `snouty validate test/antithesis/config` brings the stack up healthy, observes `setup_complete`,
  and discovers the workload test commands.
- `cargo check` (default features) is unaffected; `cargo fmt --check` is clean.

CI notes (please action before merge):
- This adds optional deps to `Cargo.lock`, so `check-licenses` (`make check-licenses`) will fail
  until `LICENSE-3rdparty.csv` is regenerated (`make license`).
- Worth confirming `check-deny` and `check-unused-deps` are happy with the feature-gated
  `antithesis_sdk` / `antithesis-instrumentation` deps.

- The research and property catalog this harness targets live in `test/antithesis/scratchbook/`
  (the next PR in this stack).
- Antithesis Rust SDK and coverage-instrumentation documentation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/test All things testing: unit/integration, correctness, SMP regression, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant