Skip to content

chore: stabilize vrt in pipeline #2115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
spliffone wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

chore: stabilize vrt in pipeline #2115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
54189ef
chore: stabilize vrt in pipeline
spliffone May 29, 2026
b4f9a98
fix(e2e): disable animations before settle-wait for infinite-animatio…
spliffone May 29, 2026
58a1a09
chore: squash
spliffone May 29, 2026
d953485
fix(filtered-search): size active criterion input to its content
spliffone May 31, 2026
2b81025
fix(filtered-search): keep free-text input readable with many criteria
spliffone May 31, 2026
17aa541
fix(e2e): revert software rendering and fix remaining VRT flakes
spliffone May 31, 2026
c120f15
fix(ci): handle non-zero exit from playwright update-snapshots in VRT…
spliffone May 31, 2026
e39489a
test(snapshots): update linux baselines after VRT stabilization changes
spliffone May 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/vrt-update.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,10 +163,12 @@ jobs:
fi
mkdir -p playwright/snapshots
echo "Starting fresh snapshot generation..."
npx playwright test --update-snapshots=all
npx playwright test --update-snapshots=all || true
else
echo "Updating only changed snapshots..."
npx playwright test --update-snapshots=changed
# playwright exits 1 when it updates snapshots (tests "fail" with diff);
# that is the expected outcome here - subsequent steps must still run.
npx playwright test --update-snapshots=changed || true
fi

# Verify that snapshots were generated
Expand Down
9 changes: 7 additions & 2 deletions playwright.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,13 @@ const config: PlaywrightTestConfig = {
forbidOnly: isCI,
/* Retry on CI only */
retries: isCI ? 1 : 0,
/* Use fixed number of workers. */
workers: 4,
/**
* Use fixed number of workers. GitHub-hosted ubuntu runners only have 2 vCPUs, so running many
* parallel browser workers saturates the CPU and makes rasterization/antialiasing timing-dependent,
* which produces flaky visual snapshots. Keep CI conservative (sharding already provides
* cross-job parallelism) and use more workers locally.
*/
workers: isCI ? 2 : 4,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
reporter: isCI
? [['dot'], ['blob']]
Expand Down
4 changes: 4 additions & 0 deletions playwright/e2e/element-examples/si-filtered-search.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,10 @@ test.describe('filtered search', () => {
await expect(page.getByRole('option', { name: 'Karlsruhe' })).toHaveClass(/active/);
await page.keyboard.type('annover');
await expect(page.getByRole('option').first()).not.toBeVisible(); // Ensures that the view was updated by Angular after typing.
// Guard against dropped keystrokes under CPU load: confirm full value before committing.
await expect(
page.locator('.pill-group', { hasText: 'Location' }).getByRole('combobox')
).toHaveValue('Hannover');
await page.keyboard.press('Enter');
await expect(freeTextSearch).toBeFocused();
await freeTextSearch.fill('Building:House');
Expand Down
14 changes: 8 additions & 6 deletions playwright/e2e/element-examples/static.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ test('si-dashboard/si-list-widget-body', ({ si }) => si.static({ delay: 2000 }))
test('si-dashboard/si-list-widget', ({ si }) => si.static({ delay: 2000 }));
test('si-dashboard/si-timeline-widget', ({ si }) => si.static({ delay: 2000 }));
test('si-dashboard/si-timeline-widget-body', ({ si }) => si.static({ delay: 2000 }));
test('si-dashboard/si-timeline-widget-css', ({ si }) => si.static());
test('si-dashboard/si-value-widget', ({ si }) => si.static());
test('si-dashboard/si-timeline-widget-css', ({ si }) => si.static({ disableAnimations: true }));
test('si-dashboard/si-value-widget', ({ si }) => si.static({ disableAnimations: true }));
test('si-datepicker/si-timepicker', ({ si }) => si.static());
test('si-electron-titlebar/si-electron-titlebar', ({ si }) => si.static());
test('si-electron-titlebar/si-fixed-height-layout-side-panel', ({ si }) => si.static());
Expand Down Expand Up @@ -78,13 +78,15 @@ test('si-landing-page/si-landing-page-single-sign-on-login', ({ si }) =>
test('si-language-switcher/si-language-switcher', ({ si }) => si.static());
test('si-layouts/content-full-layout-fixed-height', ({ si }) => si.static());
test('si-layouts/content-tile-layout-full-scroll', ({ si }) => si.static());
test('si-loading-spinner/si-loading-spinner', ({ si }) => si.static({ maxDiffPixels: 31 }));
test('si-loading-spinner/si-loading-spinner', ({ si }) =>
si.static({ maxDiffPixels: 31, disableAnimations: true }));
test('si-navbar-vertical/si-navbar-vertical-text', ({ si }) => si.static());
test('si-ncharts/si-micro-charts', ({ si }) => si.static());
test('si-pagination/si-pagination', ({ si }) => si.static());
test('si-phone-number-input/si-phone-number-input', ({ si }) => si.static());
test('si-progressbar/si-progressbar-dynamic', ({ si }) => si.static());
test('si-result-details-list/si-result-details-list', ({ si }) => si.static({ maxDiffPixels: 30 }));
test('si-result-details-list/si-result-details-list', ({ si }) =>
si.static({ maxDiffPixels: 30, disableAnimations: true }));
test('si-search-bar/si-search-bar-value', ({ si }) => si.static());
test('si-search-bar/si-search-bar', ({ si }) => si.static());
test('si-slider/si-slider', ({ si }) => si.static());
Expand All @@ -111,9 +113,9 @@ test('typography/display-styles', ({ si }) => si.static());
test('typography/typography', ({ si }) => si.static());
test('si-markdown-renderer/si-markdown-renderer', ({ si }) =>
si.static({ disabledA11yRules: ['link-in-text-block'] }));
test('si-chat-messages/si-ai-message', ({ si }) => si.static());
test('si-chat-messages/si-ai-message', ({ si }) => si.static({ disableAnimations: true }));
test('si-chat-messages/si-user-message', ({ si }) => si.static());
test('si-chat-messages/si-chat-message', ({ si }) => si.static());
test('si-chat-messages/si-chat-message', ({ si }) => si.static({ disableAnimations: true }));
test('si-chat-messages/si-attachment-list', ({ si }) => si.static());
test('si-chat-messages/si-chat-input', ({ si }) => si.static());
test('si-chat-messages/si-chat-container', ({ si }) => si.static());
Expand Down
192 changes: 48 additions & 144 deletions ...shots/ag-grid-infinite.spec.ts-snapshots/ag-grid--ag-grid-infinite--ag-grid-infinite.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,166 +8,70 @@
- columnheader "Age"
- columnheader "Company"
- rowgroup:
- row /Press Space to toggle row selection \(unchecked\) 0 Max Meier 0 Engineer \d+ years Great Company 0/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "0"
- gridcell "Max Meier 0"
- gridcell "Engineer"
- gridcell /\d+ years/
- gridcell "Great Company 0"
- row /Press Space to toggle row selection \(unchecked\) 1 Max Meier 1 Installer \d+ years Great Company 1/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "1"
- gridcell "Max Meier 1"
- gridcell "Installer"
- gridcell /\d+ years/
- gridcell "Great Company 1"
- row /Press Space to toggle row selection \(unchecked\) 2 Max Meier 2 Manager \d+ years Great Company 2/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "2"
- gridcell "Max Meier 2"
- gridcell "Manager"
- gridcell /\d+ years/
- gridcell "Great Company 2"
- row /Press Space to toggle row selection \(unchecked\) 3 Max Meier 3 Designer \d+ years Great Company 3/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "3"
- gridcell "Max Meier 3"
- gridcell "Designer"
- gridcell /\d+ years/
- gridcell "Great Company 3"
- row /Press Space to toggle row selection \(unchecked\) 4 Max Meier 4 Analyst \d+ years Great Company 4/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "4"
- gridcell "Max Meier 4"
- gridcell "Analyst"
- gridcell /\d+ years/
- gridcell "Great Company 4"
- row /Press Space to toggle row selection \(unchecked\) 5 Max Meier 5 Engineer \d+ years Great Company 0/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "5"
- gridcell "Max Meier 5"
- gridcell "Engineer"
- gridcell /\d+ years/
- gridcell "Great Company 0"
- row /Press Space to toggle row selection \(unchecked\) 6 Max Meier 6 Installer \d+ years Great Company 1/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "6"
- gridcell "Max Meier 6"
- gridcell "Installer"
- gridcell /\d+ years/
- gridcell "Great Company 1"
- row /Press Space to toggle row selection \(unchecked\) 7 Max Meier 7 Manager \d+ years Great Company 2/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "7"
- gridcell "Max Meier 7"
- gridcell "Manager"
- gridcell /\d+ years/
- gridcell "Great Company 2"
- row /Press Space to toggle row selection \(unchecked\) 8 Max Meier 8 Designer \d+ years Great Company 3/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "8"
- gridcell "Max Meier 8"
- gridcell "Designer"
- gridcell /\d+ years/
- gridcell "Great Company 3"
- row /Press Space to toggle row selection \(unchecked\) 9 Max Meier 9 Analyst \d+ years Great Company 4/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell "9"
- gridcell "Max Meier 9"
- gridcell "Analyst"
- gridcell /\d+ years/
- gridcell "Great Company 4"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Engineer \d+ years Great Company 0/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Engineer"
- gridcell /\d+ years/
- gridcell "Great Company 0"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Installer \d+ years Great Company 1/:
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Installer"
- gridcell /\d+ years/
- gridcell "Great Company 1"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Manager \d+ years Great Company 2/:
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Manager"
- gridcell /\d+ years/
- gridcell "Great Company 2"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Designer \d+ years Great Company 3/:
- gridcell
- gridcell
- gridcell
- gridcell
- gridcell
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Designer"
- gridcell /\d+ years/
- gridcell "Great Company 3"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Analyst \d+ years Great Company 4/:
- gridcell
- gridcell
- gridcell
- gridcell
- gridcell
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Analyst"
- gridcell /\d+ years/
- gridcell "Great Company 4"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Engineer \d+ years Great Company 0/:
- gridcell
- gridcell
- gridcell
- gridcell
- gridcell
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Engineer"
- gridcell /\d+ years/
- gridcell "Great Company 0"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Installer \d+ years Great Company 1/:
- gridcell
- gridcell
- gridcell
- gridcell
- gridcell
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Installer"
- gridcell /\d+ years/
- gridcell "Great Company 1"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Manager \d+ years Great Company 2/:
- gridcell
- gridcell
- gridcell
- gridcell
- gridcell
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Manager"
- gridcell /\d+ years/
- gridcell "Great Company 2"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Designer \d+ years Great Company 3/:
- gridcell
- gridcell
- gridcell
- gridcell
- gridcell
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Designer"
- gridcell /\d+ years/
- gridcell "Great Company 3"
- row /Press Space to toggle row selection \(unchecked\) \d+ Max Meier \d+ Analyst \d+ years Great Company 4/:
- gridcell
- gridcell
- gridcell
- gridcell
- gridcell
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
- gridcell /\d+/
- gridcell /Max Meier \d+/
- gridcell "Analyst"
- gridcell /\d+ years/
- gridcell "Great Company 4"
- gridcell
- gridcell
- gridcell
- gridcell
- gridcell
- row "Press Space to toggle row selection (unchecked)":
- gridcell "Press Space to toggle row selection (unchecked)":
- checkbox "Press Space to toggle row selection (unchecked)"
Expand Down
4 changes: 2 additions & 2 deletions ...-search-playground--invalid-criterion-element-examples-chromium-light-linux.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion ...ec.ts-snapshots/si-filtered-search--si-filtered-search-playground--invalid-criterion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
- text: Filtered Search Location Hannover
- text: Filtered Search Location Ha
- button "clear"
- text: Building House
- button "clear"
Expand Down
4 changes: 2 additions & 2 deletions ...ered-search-playground--operator-open-element-examples-chromium-light-linux.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions ...pinner--si-loading-button--second-step-element-examples-chromium-dark-linux.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions ...inner--si-loading-button--second-step-element-examples-chromium-light-linux.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions ...c.ts-snapshots/list-group--list-group-element-examples-chromium-light-linux.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions ...ts/si-chat-messages--si-chat-container-element-examples-chromium-dark-linux.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions ...s/si-chat-messages--si-chat-container-element-examples-chromium-light-linux.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
15 changes: 14 additions & 1 deletion playwright/support/test-helpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,12 @@ export type StaticTestOptions = {
waitCallback?: (page: Page) => Promise<void>;
skipAutoScaleViewport?: boolean;
skipAriaSnapshot?: boolean;
/**
* Disables CSS transitions/animations before waiting for animations to settle. Required for
* examples with infinite animations (e.g. loading spinners, shimmer/typing indicators) where
* `document.getAnimations()` never drains to zero and the settle-wait would otherwise time out.
*/
disableAnimations?: boolean;
};

// Playwright since 1.48 has the mouse cursor at 0/0 causing any element at this coordinate to be
Expand Down Expand Up @@ -108,6 +114,10 @@ class SiTestHelpers {
if (options?.delay) {
await this.page.waitForTimeout(options?.delay);
}
if (options?.disableAnimations) {
await this.enableDisableAnimations(this.page, false);
}
await this.waitForAllAnimationsToComplete();
await this.runVisualAndA11yTests(step, {
axeRulesSet: options?.disabledA11yRules?.map(item => ({ id: item, enabled: false })),
maxDiffPixels: options?.maxDiffPixels,
Expand Down Expand Up @@ -233,12 +243,14 @@ class SiTestHelpers {
}
if (this.testInfo.project.metadata.isVrt) {
try {
await this.enableDisableAnimations(this.page, false);
await this.showHideIgnores(this.page, false, options?.snapshotDelay);
await expect(this.page).toHaveScreenshot(testName + '.png', {
maxDiffPixels: options?.maxDiffPixels
});
} finally {
await this.showHideIgnores(this.page, true);
await this.enableDisableAnimations(this.page, true);
}

if (!options?.skipAriaSnapshot && !this.testInfo.project.metadata.skipAriaSnapshot) {
Expand All @@ -255,7 +267,8 @@ class SiTestHelpers {
public async waitForAllAnimationsToComplete(threshold = 0): Promise<void> {
await this.page.waitForFunction(
count => window.document.getAnimations().length <= count,
threshold
threshold,
{ timeout: 5000 }
);
Comment on lines 268 to 272
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot May 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If the page contains any infinite animations (such as loading spinners, infinite progress bars, or animated icons), window.document.getAnimations() will always return a non-empty array. This will cause waitForAllAnimationsToComplete to consistently time out after 5000ms and fail the test.

To prevent this, we should filter out infinite animations (where iterations is Infinity) so that we only wait for finite transitions and animations to complete.

    await this.page.waitForFunction(
      count => window.document.getAnimations().filter(anim => {
        const iterations = anim.effect?.getTiming().iterations;
        return iterations !== Infinity;
      }).length <= count,
      threshold,
      { timeout: 5000 }
    );

}
/**
Expand Down
Loading
Loading