Run E2E tests against the latest stable version of Gutenberg

[fullofcaffeine]

Is your feature request related to a problem? Please describe.

Yes. Changes in Gutenberg can break CoBlocks and vice-versa. Here's a recent example of a bug that happens when a recent version of CoBlocks is activated alongside a relatively recent version of Gutenberg: #2474.

Since CoBlocks extends and depends on Gutenberg, changes in Gutenberg can affect CoBlocks, and changes in CoBlocks could also affect Gutenberg.

Describe the solution you'd like

Add a new job to test against the latest stable version of Gutenberg in PRs and/or recurringly. I'm thinking about the E2E tests here.

I'm not too acquainted with the CoBlocks source, but I assume that E2Es are running against the built-in block editor that comes with WPorg, which is usually equivalent to a pretty old version of Gutenberg.

Thanks!

[olafleur-godaddy]

Yes, you are correct, we run our e2e pipeline on the latest version of WordPress, without Gutenberg. We usually have a branch on the side on which we test for WP beta, RCs and with Gutenberg, which drives our compatibility work to make sure that when the Final version releases, we fixed all the problems we see.

The issue that I see in putting that in our build pipeline is that it would be failing often, out of our control. Our goal is to have the pipeline always green if possible. But yeah, I think something recurring or with a report somehow would be interesting.

@jkopepasah-godaddy, any ideas on the subject?

[kopepasah]

Hello @fullofcaffeine, and thanks for the detailed thoughts. Because latest of Gutenberg can often include changes not yet fully flushed out, we are a bit hesitant to started testing against trunk. This boils down to signal vs. noise in our tests, and if we are running tests against code that could very-well change in the next trunk push, it could lead us down a road of debugging a problem that may fix itself.

I am not against writing some tests against versions of Gutenberg; however, Gutenberg itself is non-production software, and thus we would be running tests against software considered "in development", which is not the best case for running tests, as these could sometimes be very unreliable.

Now, from the issue you mentioned, it looks like this is occurring in WP 6.1.1, as well as the Gutenberg plugin. So we likely need to write a test to cover this instance.

Lastly, we just want to note that currently our tests are running in Cypress, but much like Gutenberg, we are currently under a migration to Playwright, so it may be that these tests do not make it in until that migration is complete (just FYI).

[kopepasah]

@fullofcaffeine actually, I stand corrected, as this only seems to appear with the latest Gutenberg plugin installed. We will continue to keep an eye on this change to determine what fix will be required in CoBlocks, once this nears a release to the production version of WordPress Block Editor.

[WunderBart]

👋 @kopepasah I've opened a PR fixing the aforementioned issue (#2474) - it should work with the latest Gutenberg with a fallback to the trunk version if the experimental feature is unavailable. I'd appreciate it if you could take a look! Let me know if you'd like me to add some tests there. 🙌