Skip to content

fix(openfeature): make InitWithContext return ready without a first config#5053

Draft
vjfridge wants to merge 1 commit into
mainfrom
vickie/ffl-2809-openfeature-init-empty-config
Draft

fix(openfeature): make InitWithContext return ready without a first config#5053
vjfridge wants to merge 1 commit into
mainfrom
vickie/ffl-2809-openfeature-init-empty-config

Conversation

@vjfridge

Copy link
Copy Markdown

What

Change openfeature.DatadogProvider.InitWithContext so it does not block indefinitely waiting for the first Remote Config payload. Init now performs a short, bounded opportunistic wait (initialConfigWait = 5s) for the first config, and returns nil (ready) whether or not that config arrived — flag evaluations pre-config resolve to caller-supplied defaults with reason DEFAULT and pick up the real config as soon as the RC callback fires.

Why

InitWithContext previously looped on configuration != nil, waiting on configChange.Broadcast and bounded only by the caller's ctx (or Init's own 30s timeout when called without a ctx). That treated "the agent has not yet delivered a config for FFE_FLAGS" as an initialization failure, but it's a legitimate steady state:

  • No configs are targeted at this service (nothing published, or targeting excludes it).
  • The initial poll response omits the product because the sidecar's cache is cold for the pod's client-id.

The RC subscription being established is a stronger readiness signal than a config arriving — the subscription is a precondition for reaching InitWithContext at all (see startWithRemoteConfig calling SubscribeProvider / AttachCallback synchronously).

Observed failure on ffe-service in staging (gizmo): every DPA-driven vertical in-place resize triggered a main-container restart (staging-only rapid change 6af559ea4e02d set resizePolicy: RestartContainer for services with vertical autoscaling). The new Go process re-entered InitWithContext; the sidecar's response for that fresh subscription contained no FFE_FLAGS payload; configChange.Broadcast never fired; the container was killed by the startup probe at 30s. Logs on the crashing pods show "openfeature-datadog: Init called" at t=0, "openfeature-datadog: Init completed successfully" never, and the container exiting exactly 30s later — repeated across every restart cycle until the DPA was patched to Preview.

How

  • InitWithContext wraps ctx in context.WithTimeout(ctx, initialConfigWait) and calls waitForConfigurationUpdate once. Any error (deadline, cancel) is intentionally swallowed.
  • Added initialConfigWait = 5s — long enough to cover a normal PollInterval-bounded delivery on the happy path, short enough that a service with an idle RC feed still comes ready inside a typical startup-probe budget, well below defaultInitTimeout so callers passing context.Background() don't collide with an outer probe deadline.

Existing behavior preserved:

  • If the caller's ctx deadline is shorter than initialConfigWait, we honor the caller's deadline.
  • If a config arrives during the wait, Init returns immediately with the config loaded (see TestSetProviderWithContextAndWaitConfigArrivesDuringInit).
  • Exposure writer + EVP flag-eval writer still start.
  • Init() (no-ctx form) still uses defaultInitTimeout as its outer bound; the change is what happens within that budget.

Tests

  • Renamed TestSetProviderWithContextAndWaitTimeoutTestSetProviderWithContextAndWaitNoConfig. Old test asserted the buggy behavior (Init returns context.DeadlineExceeded when no config arrives). New test asserts Init returns nil inside the caller's deadline.
  • Added TestSetProviderWithContextAndWaitConfigArrivesDuringInit — Init unblocks immediately when a config arrives during the opportunistic wait, and the config is visible on the provider afterward.
  • Full ./openfeature/... and ./internal/openfeature/... suites pass with -race.

Follow-ups (out of scope)

  • The configChange.Wait goroutine started by waitForConfigurationUpdate is not cancelable by ctx — it only exits when a broadcast eventually fires. Pre-existing, not introduced by this PR. Because Init runs at most once per process, this leaks at most one goroutine until the first config eventually arrives; acceptable but worth cleaning up separately.
  • The dd-source-side rapid framework change that enabled in-place-restart on staging (resizePolicy: RestartContainer) should be reviewed jointly — any service with a startup-probe budget ≤ 30s that depends on RC-delivered config at startup is exposed to the same class of failure. This PR removes the OpenFeature-specific exposure; a broader audit of what else blocks on RC at startup is the Rapid team's call.

Internal tracking: FFL-2809.

…onfig

InitWithContext previously blocked on `configuration != nil` until the
caller's ctx expired or the internal 30s timeout fired. That was wrong:
the provider is ready as soon as its RC subscription is established
(SubscribeProvider / AttachCallback in startWithRemoteConfig run
synchronously before Init), not when the first config payload arrives.

When the agent's initial RC response for a subscribed FFE_FLAGS client
omits the product — either because no configs are targeted at the
service, or because a container restart lands during a poll window where
the sidecar has nothing cached — no callback ever fires. Init then
consumed its full deadline. Combined with the recent rapid framework
change setting resizePolicy: RestartContainer on staging services with
vertical autoscaling (dd-source 6af559ea4e02d), every DPA-driven
in-place resize started re-triggering the wait and blowing past the
30s startup-probe budget, sending ffe-service pods into CrashLoopBackOff
on every applied recommendation.

Init now waits opportunistically up to a bounded initialConfigWait (5s)
so fresh deployments still block briefly for real flag data on the
happy path, but returns successfully with no error if the wait expires
or the caller's ctx is cancelled first. Flag evaluations before the
first config resolve to caller-supplied defaults with reason DEFAULT;
when the config arrives via the RC callback, evaluations pick it up
without any restart or reinit.

Updated TestSetProviderWithContextAndWaitTimeout — which encoded the
old buggy behavior — into two tests: one asserting Init returns without
error when no config arrives and honors the caller ctx, one asserting
Init unblocks as soon as a config arrives during the wait.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Jul 21, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 62.85%

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 18ba72f | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jul 21, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-21 17:12:45

Comparing candidate commit 18ba72f in PR branch vickie/ffl-2809-openfeature-init-empty-config with baseline commit 2c9ad82 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 325 metrics, 1 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:BenchmarkOTLPTraceWriterFlush

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant