Skip to content

TEST ONLY: SymDB extractor without throttle (regression detection check for #5776)#5784

Draft
p-datadog wants to merge 1 commit into
masterfrom
ddsign/symdb-extractor-no-throttle-test
Draft

TEST ONLY: SymDB extractor without throttle (regression detection check for #5776)#5784
p-datadog wants to merge 1 commit into
masterfrom
ddsign/symdb-extractor-no-throttle-test

Conversation

@p-datadog
Copy link
Copy Markdown
Member

@p-datadog p-datadog commented May 19, 2026

JIRA: DEBUG-5668

Summary

Do not merge. This is a sibling-branch test for #5776.

Branches from #5776's tip and reverts only the throttle implementation in lib/datadog/symbol_database/extractor.rb + sig/datadog/symbol_database/extractor.rbs. The benchmarks and helper additions from #5776 are kept.

The point is to verify that the benchmarks added in #5776 actually catch a throttle regression, by running CI against a branch where the throttle has been removed.

What's removed vs #5776's tip

  • SLEEP_EVERY_N_MODULES = 100 constant
  • SLEEP_SECONDS = 0.001 constant
  • private_constant :SLEEP_EVERY_N_MODULES, :SLEEP_SECONDS
  • The sleep SLEEP_SECONDS if (seen % SLEEP_EVERY_N_MODULES).zero? call (and its seen counter) in collect_extractable_modules
  • Matching RBS lines

What to check in CI

Compare CI output here vs #5776:

  1. symbol_database_background_impactp99_ratio_treatment_over_baseline should jump from ~1.05 (PR 5776 with throttle) to well above 1.5 (this PR). The in-script enforce_requirement gate at threshold 1.50 should fire exit 1.
  2. symbol_database_baseline_matrix — the 100%-arm rps_drop should jump from ~8% to materially higher. Lower-baseline arms are noisier in microbenchmark form (documented limitation), but should also shift.
  3. bp-runner microbenchmarks (gitlab) — large ops/sec drop on the workload arms of both benchmarks vs master.
  4. symbol_database_extractionwall_time_seconds should drop ~2x (no per-chunk sleep) and cpu_percent should approach 100%. Memory should be unchanged.

Cleanup

Delete this branch once the comparison has been recorded.

Change log entry

None.

@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label May 19, 2026
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented May 19, 2026

👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description.

If changes need to be present in CHANGELOG.md you can state it this way

**Change log entry**

Yes. A brief summary to be placed into the CHANGELOG.md

(possible answers Yes/Yep/Yeah)

Or you can opt out like that

**Change log entry**

None.

(possible answers No/Nope/None)

Visited at: 2026-05-19 22:18:21 UTC

@dd-octo-sts dd-octo-sts Bot added the dev/testing Involves testing processes (e.g. RSpec) label May 19, 2026
@datadog-prod-us1-4
Copy link
Copy Markdown

datadog-prod-us1-4 Bot commented May 19, 2026

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-rb | microbenchmarks: [other]   View in Datadog   GitLab

🔧 Fix in code (Fix with Cursor). TypeError: string indices must be integers in conversion process, causing benchmark analysis to fail.

Check Pull Request CI Status | all-jobs-are-green   View in Datadog   GitHub Actions

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 97.08%

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c70a393 | Docs | Datadog PR Page | Give us feedback!

DO NOT MERGE. Sibling-branch test for PR #5776: removes the
SLEEP_EVERY_N_MODULES / SLEEP_SECONDS throttle from
collect_extractable_modules to verify that the benchmarks added in
#5776 detect the regression bp-runner is supposed to gate.

Expected signals on CI:
  - symbol_database_background_impact: p99 ratio
    treatment_over_baseline jumps from ~1.05 to >1.5
  - symbol_database_baseline_matrix: 100%-arm rps_drop jumps from
    ~8% to ~25%+; lower-baseline arms may also shift
  - bp-runner: significant ops/sec drop on workload arms of both
    benchmarks vs master (master here = PR #5776 tip with throttle
    intact)

If the throttle is later restored to master, this branch can be
deleted.
@p-datadog p-datadog force-pushed the ddsign/symdb-extractor-no-throttle-test branch from 3dd7ca2 to c70a393 Compare May 20, 2026 17:15
@dd-octo-sts dd-octo-sts Bot added the debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database) label May 20, 2026
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented May 20, 2026

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 6 steep:ignore comments, and clears 6 steep:ignore comments.

steep:ignore comments (+6-6)Introduced:
lib/datadog/symbol_database/extractor.rb:659
lib/datadog/symbol_database/extractor.rb:723
lib/datadog/symbol_database/extractor.rb:724
lib/datadog/symbol_database/extractor.rb:804
lib/datadog/symbol_database/extractor.rb:805
lib/datadog/symbol_database/extractor.rb:853
Cleared:
lib/datadog/symbol_database/extractor.rb:678
lib/datadog/symbol_database/extractor.rb:742
lib/datadog/symbol_database/extractor.rb:743
lib/datadog/symbol_database/extractor.rb:823
lib/datadog/symbol_database/extractor.rb:824
lib/datadog/symbol_database/extractor.rb:872

Untyped methods

This PR introduces 1 untyped method and 7 partially typed methods, and clears 1 untyped method and 7 partially typed methods.

Untyped methods (+1-1)Introduced:
sig/datadog/symbol_database/extractor.rbs:13
└── def initialize: (logger: untyped, settings: untyped) -> void
Cleared:
sig/datadog/symbol_database/extractor.rbs:15
└── def initialize: (logger: untyped, settings: untyped) -> void
Partially typed methods (+7-7)Introduced:
sig/datadog/symbol_database/extractor.rbs:37
└── def build_class_language_specifics: (Class klass) -> Hash[::Symbol, untyped]
sig/datadog/symbol_database/extractor.rbs:51
└── def collect_extractable_modules: () -> Hash[String, untyped]
sig/datadog/symbol_database/extractor.rbs:53
└── def group_methods_by_file: (Module mod) -> Hash[String, Array[untyped]]
sig/datadog/symbol_database/extractor.rbs:55
└── def build_file_trees: (Hash[String, untyped] entries) -> Hash[String, untyped]
sig/datadog/symbol_database/extractor.rbs:57
└── def place_in_tree: (untyped root, Array[String] name_parts, Module mod, String mod_name, Array[untyped] methods, String file_path) -> void
sig/datadog/symbol_database/extractor.rbs:61
└── def convert_trees_to_scopes: (Hash[String, untyped] file_trees) -> Array[Scope]
sig/datadog/symbol_database/extractor.rbs:63
└── def convert_node_to_scope: (untyped node) -> Scope
Cleared:
sig/datadog/symbol_database/extractor.rbs:39
└── def build_class_language_specifics: (Class klass) -> Hash[::Symbol, untyped]
sig/datadog/symbol_database/extractor.rbs:53
└── def collect_extractable_modules: () -> Hash[String, untyped]
sig/datadog/symbol_database/extractor.rbs:55
└── def group_methods_by_file: (Module mod) -> Hash[String, Array[untyped]]
sig/datadog/symbol_database/extractor.rbs:57
└── def build_file_trees: (Hash[String, untyped] entries) -> Hash[String, untyped]
sig/datadog/symbol_database/extractor.rbs:59
└── def place_in_tree: (untyped root, Array[String] name_parts, Module mod, String mod_name, Array[untyped] methods, String file_path) -> void
sig/datadog/symbol_database/extractor.rbs:63
└── def convert_trees_to_scopes: (Hash[String, untyped] file_trees) -> Array[Scope]
sig/datadog/symbol_database/extractor.rbs:65
└── def convert_node_to_scope: (untyped node) -> Scope

Untyped other declarations

This PR introduces 2 untyped other declarations, and clears 2 untyped other declarations. It decreases the percentage of typed other declarations from 82.48% to 82.46% (-0.02%).

Untyped other declarations (+2-2)Introduced:
sig/datadog/symbol_database/extractor.rbs:11
└── @logger: untyped
sig/datadog/symbol_database/extractor.rbs:12
└── @settings: untyped
Cleared:
sig/datadog/symbol_database/extractor.rbs:13
└── @logger: untyped
sig/datadog/symbol_database/extractor.rbs:14
└── @settings: untyped

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database) dev/testing Involves testing processes (e.g. RSpec)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants