Spike - evaluate integration opportunity

[blackfalcon]

Discussed in #495

^{Originally posted by blackfalcon February 5, 2024}
Alaska has invested into a series of a11y testing tools from LevelAcess which include CI/CD pipeline delivery automated testing.

Here is a link for the JavaScript SDK https://client.levelaccess.com/hc/en-us/articles/13844935132439-Get-started-with-Access-Continuum#JavaScript-SDK

Auro web components use the Modern Web test runner tools and here is information on using Playwrite https://modern-web.dev/docs/test-runner/testing-in-a-ci/#playwright and is stated as being supported by LevelAccess https://client.levelaccess.com/hc/en-us/articles/13844879206807-Access-Continuum-overview

The goal of this discussion is to understand the lift needed for using LevelAccess automated testing for all web components to ensure a high quality of accessibility with each component.

LevelAccess does have demo projects, but Playwrite is not listed. I will be checking in LevelAccess to see about support.
https://alaskaair.hub.essentia11y.com/integrations/test-automation?linkedPropertyData=652813da6eec28e13de119c4%7C652813da6eec28ef1ee119bd

Scope

The scope of this issue is to review the information above and perform a spike on level of effort needed to fully integrate the LevelAccess CI/CD testing tools.

Exit criteria

This issue can be closed once the Spike has been completed and there is a full evaluation of the integration effort with possibly an example branch with any demo or currently existing element.

If it is decided to move forward with full integration, we need a new issue created with the scope and exit criteria.

[fajar-apri-alaska]

Hi team,
So, I was researching Continuum (from LevelAccess) SDK with Lit components for the following week.
At first, I did it with auro-carousel that still using karma test runner. I just follow the instruction from the LevelAccess documentation, it does works with karma, I could saw the accessibility concerns got printed to the console, but sad news that my main branch was still not updated yet. 😞

Next, the web-test-runner(WTR) that Auro currently uses, the SDK couldn't be injected the same way as karma does.
So that's where Playwright comes in, I then did my research again.
Playwright next, has an experimental feature to test components.
It exposes mount hooks to the test enviroment, where it does render with an html file that will be used to render the components during testing.

So the idea is, inject the Continuum SDK in the html used for testing environment, to expose Continuum global script, which later we will setup the SDK, and call the runAllTests() function based on this documentation, then we should see the accessibility concerns from the SDK. (like it does work in karma)

continuum.runAllTests().then((accessibilityConcerns) => {
    // you can use either accessibilityConcerns or invoke continuum.getAccessibilityConcerns() here;
    // both will return the same data: any accessibility concerns that were found on the page during this last test run
});

The implementation though, sure are hitting a wall. Official Playwright doesn't have support for lit-element.
Someone already posted an issue for the support.
There is also someone working with the issue and provides an alternative dependency to support lit-element.
But again, I did hit another wall, the issue are the same exact as this one, the Author is currently working with that also.

So for now, I am currently leaving my trace there to see the updates.

[blackfalcon]

@fajar-apri-alaska you are correct. Playwrite does not support Lit straight out of the box. This is where I am assuming npm i @web/test-runner will need to be used. https://modern-web.dev/docs/test-runner/browser-launchers/playwright/#testing-multiple-browsers

There is a meeting scheduled with LevelAccess to better understand their integration with Playwrite and I am hoping that this will unlock some potential in this area.