Fixed flaky configOverrideSection integration test timeout in CI#1514
Draft
carterworks wants to merge 9 commits into
Draft
Fixed flaky configOverrideSection integration test timeout in CI#1514carterworks wants to merge 9 commits into
carterworks wants to merge 9 commits into
Conversation
vitest 4.1.5 requires the browser provider to be configured via a factory in the vitest config rather than a CLI string. The config already uses the playwright() factory from @vitest/browser-playwright.
|
…ewthrough_ids test
…uld pass first time
The earlier commit assumed react-spectrum overlay animations were what
was blowing playwright's actionability budget in CI. They aren't:
react-spectrum's Popover uses <Transition timeout={{enter: 0, exit: 350}}>
so the enter animation is already instantaneous and the exit doesn't
gate actionability checks. The real cause is React commit latency under
CI CPU load when a Formik field update conditionally mounts the next
field in the test's sequence; that's being fixed at the test level.
…it latency in CI
In the configOverrideSection 'updates form values and saves to settings'
test, each selectOption('Enabled') conditionally mounts the next field
via Formik. Under CI CPU load the next field isn't fully committed by
React when the test tries to interact with it, blowing the 2s
actionability budget and burning retries until the 30s test timeout
fires. Wait for the field to be visible before opening its dropdown,
matching the existing pattern used for experiencePlatformEnabled.
…ate leakage
browser/integration runs with isolate: false + fileParallelism: true,
so all advertising test files share one browser context and one cookie
jar. Once any test fires markIdsAsConverted (via handleViewThrough),
the resulting \${idType}_last_conversion entries persist for 30 minutes
inside the kndctr_<orgId>_advertising cookie. Subsequent runs of
viewthrough_ids.spec.js see SURFER_ID as throttled, which both
short-circuits collectAllIdentities (no enrichment call sent) and
trips onBeforeSendEvent's early-return (no query.advertising on the
sendEvent), making it structurally impossible for the test to find a
view-through call.
Clean up alloy-namespaced cookies in cleanAlloy() so each test starts
from a fresh state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changed Packages
Description
Fixed an intermittent test timeout in the
configOverrideSectionintegration test suite. The failing test (Config overrides section > updates form values and saves to settings) was hitting the 30 s vitest timeout in CI while passing locally.Root cause: Under headless CI conditions, playwright's actionability checks wait for react-spectrum dropdown overlay animations to fully settle before registering a click. The
selectOption()helper has a 5 s click timeout × 3 retries = up to 15 s per call. With multipleselectOption()calls in sequence, total accumulated wait time could exceed the 30 s test timeout before the retry logic even had a chance to throw an error.Fix:
<style>block in the integration test setup that setsanimation-duration,animation-delay,transition-duration, andtransition-delayto0sfor all elements. Dropdown overlays that previously animated in over ~200 ms (slower under CI load) now appear instantly, making playwright actionability checks succeed immediately.retry: 2to the vitestreactor-extension/integrationproject config as a safety net for any remaining transient flakiness.Also removed the deprecated
--browser.provider=playwrightCLI flag from thetest:unit:debugandtest:integration:debugnpm scripts — the provider is already fully configured invitest.projects.jsand the flag was a no-op after recent vitest upgrades.Related Issue
CI failure: https://github.com/adobe/alloy/actions/runs/26183634291/job/77033100029
Motivation and Context
This test was observed failing in CI with
Error: Test timed out in 30000ms. The flakiness is timing-dependent and only reproduced under headless CI conditions, not locally in headed mode.Types of changes
Checklist: