elements@1 - Auto-load custom-elements (web-components) from extensions

[totten]

Overview

Define a file-naming convention for "Custom Elements" (aka "Web Component"s) based on *.js, *.mjs, and/or *.css files. Implement lazy-loading with ECMAScript module (ESM) support.

(Ping @ufundo @colemanw. This is a re-spin of #34325 which adds CSS support, changes the name from component-js to just elements, and adds some test-coverage.)

Technical Details: Consumers

As a consumer of a custom-element, you simply use the tag... somewhere. It can be Quickform or AngularJS or jQuery or whatever. For example:

<hello-world></hello-world>

But how is this new tag defined?

Technical Details: Providers

As the provider of a custom element, you will do the following:

1. Update your info.xml and enable <mixin>elements@1</mixin>.
2. Create a file in your extension called element/TAG-NAME.js.

For example, here is an implementation of <hello-world>:

// FILE: element/hello-world.js

class HelloWorld extends HTMLElement {
  connectedCallback() {
    this.textContent = 'Hello world!';
  }
}

customElements.define('hello-world', HelloWorld);

Since this supports ECMAScript modules, you can import helpers from other files, as in:

import { helper } from 'org.civicrm.otherext/js/otherHelper.js';

// Note: To have this kind of sharing between the extensions, it's best if both use elements@1.

This is implemented as a mixin, and it should be amenable for backporting on 5.63+.

Comments

The general approach is this:

• Use AssetBuilder to create a semi-static file, elements.js, which stores an index of all these CustomElements.
• On every pageload, include this index file.
• Use MutationObserver to determine when new elements are needed.
• This index shouldn't be too big (at least, with our kind of usage/scale). For example, with 100 CustomElements, the index should be ~10kb. (The indexing mechanism is encapsulated, so it can be changed if size becomes an issue.)
• There is one MutationObserver shared by all these CustomElements.

Limitations worth considering:

• It uses lazy-loading (i.e. on-demand). This means you may perceive some flashing/delay as components render.

  (I imagine some convention could help manage that... but I'm not sure what such a convention should be...)

• This doesn't specifically implement bundling, and we can defer that question. However, experiments based on this technique are promising. The key thing is that the registry maps ['my-element' => [...jsFile, cssFile...]]. This is many-to-one. Multiple elements can map to the same file, and the file is only be loaded once. So it should be agreeable to various tricks/techniques (with trade-offs on techniques).

• Translations - We probably need some more logic to pull in strings for these import()d JS files.

[civibot]

🤖 Thank you for contributing to CiviCRM! ❤️ We will need to test and review this PR. 👷

Introduction for new contributors...

• If this is your first PR, an admin will greenlight automated testing with the command ok to test or add to whitelist.
• A series of tests will automatically run. You can see the results at the bottom of this page (if there are any problems, it will include a link to see what went wrong).
• A demo site will be built where anyone can try out a version of CiviCRM that includes your changes.
• If this process needs to be repeated, an admin will issue the command test this please to rerun tests and build a new demo site.
• Before this PR can be merged, it needs to be reviewed. Please keep in mind that reviewers are volunteers, and their response time can vary from a few hours to a few weeks depending on their availability and their knowledge of this particular part of CiviCRM.
• A great way to speed up this process is to "trade reviews" with someone - find an open PR that you feel able to review, and leave a comment like "I'm reviewing this now, could you please review mine?" (include a link to yours). You don't have to wait for a response to get started (and you don't have to stop at one!) the more you review, the faster this process goes for everyone 😄
• To ensure that you are credited properly in the final release notes, please add yourself to contributor-key.yml
• For more information about contributing, see CONTRIBUTING.md.

Quick links for reviewers...

• ➡️ PR Demo site (can be rebuilt with the command test this please).
• 📖 Review standards
• 🗒️ Review template (brief or verbose)

➡️ Online demo of this PR 🔗

[colemanw]

@totten this looks really cool. Couple questions:

1. So the .css file is autoloaded if it's in the same directory and has the same name?
2. Are these files always loaded with assetBuilder? Not directly?
3. We've never used the .mjs naming convention before. Seems like it might be YAGNI, since minification is waning in popularity these days. (ignore that, resolved in https://lab.civicrm.org/dev/core/-/issues/6257)

[totten]

1. So the .css file is autoloaded if it's in the same directory and has the same name?

Correct.

2. Are these files always loaded with assetBuilder? Not directly?

• Yes, in that... the index file (the list of available tags/elements) is produced by asset-builder.
• No, in that... the JS/CSS files for each element is loaded separately.

Compare to autoloading with PHP-Composer:

composer framework	elements@1 framework
Auto-generates vendor/autoload.php via composer.phar.	Auto-generates elements.js via AssetBuilder.
autoload.php is loaded on every page-view.	elements.js is loaded on every page-view.
autoload.php is an index/listing.	elements.js is an index/listing.
Specific PHP class-files are loaded on-demand.	Specific JS element-files are loaded on-demand.
autoload.php and all the specific PHP class-files use the opcode cache.	elements.js and all the specific JS class-files use the browser cache.
When generating vendor/autoload.php, you do some scans.	When generating elements.js, you do some scans.

3. We've never used the .mjs naming convention before. Seems like it might be YAGNI, since minification is waning in popularity these days.

Minification? There may be a miscommunication. *.mjs stands for "Module JavaScript" not "Minified Javascript" (which would be *.min.js). Example reference.

The file-extension question is a squishy thing, though. I'm not saying EDNI(?) ("Everyone Definitely Needs It!"); it does merit more conversation. It's really a separate thread, though. (The support in here is "compatible with" or "prepared for" an *.mjs world, but it's not required.) Let me fork off a separate issue about *.mjs: https://lab.civicrm.org/dev/core/-/issues/6257

[colemanw]

Normally we also don't escape unicode.

civicrm-core/CRM/Utils/JSON.php

Line 45 in 6eef79a

echo json_encode($input, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);

[colemanw]

@totten cool, so how will translatable strings get extracted and sent to the client?
With other scripts, that happens here:

civicrm-core/CRM/Core/Resources/CollectionTrait.php

Lines 115 to 119 in 92a3299

	if ($snippet['translate']) {
	$domain = ($snippet['translate'] === TRUE) ? $ext : $snippet['translate'];
	// Is this too early?
	$this->addString(Civi::service('resources.js_strings')->get($domain, $res->getPath($ext, $file), 'text/javascript'), $domain);
	}

And with Angular it happens here:

civicrm-core/Civi/Angular/Manager.php

Lines 295 to 313 in 82b0e85

	public function getStrings($name) {
	$module = $this->getModule($name);
	$result = [];
	if (isset($module['js'])) {
	foreach ($module['js'] as $file) {
	$strings = $this->res->getStrings()->get(
	$module['ext'],
	$this->res->getPath($module['ext'], $file),
	'text/javascript'
	);
	$result = array_unique(array_merge($result, $strings));
	}
	}
	$partials = $this->getPartials($name);
	foreach ($partials as $partial) {
	$result = array_unique(array_merge($result, \CRM_Utils_JS::parseStrings($partial)));
	}
	return $result;
	}

[totten]

(@colemanw) cool, so how will translatable strings get extracted and sent to the client?

That's a really good question, and I'm not certain the answer.

The code-style of ESM would tend to emphasize using static imports. If you were doing pure ESM without any other tooling, you might have an adjacent data-files:

import ts from "myextension/element/hello-world.strings.js";

The strings need to vary by user locale, but you could address that by mapping the path to asset-builder. I'm nervous about increasing the #requests multiplicatively. (One JS file-request per custom-element? For (say) 30 elements, that's 30 requests... fine? But for JS+CSS+strings, then it's 90 requests. Hmm...)

Maybe a better balance is to fetch the strings per-extension or per-folder? (So search_kit has a strings-JSON; and then afform_admin has another strings-JSON).

import ts from "myextension/element/ALL-STRINGS.js"; ## Handled via asset-builder

But there are still a more angles to work out. (Caching/locale) And I kinda suspect it'll be easier to work out if we get the strings with a dynamic import, e.g.

const ts = await import(CRM.url('civicrm/ajax/strings?target=' + import.meta.resolve('.') + "&locale="+...));
const ts = await import(CRM.stringsUrl('myextension'));
const ts = await import(CRM.stringsUrl(import.meta));
const ts = await import(CRM.stringsUrl(import.meta)) with { type: "json" };
const ts = await CRM.importStrings(import.meta);

So getting toward the end there, maybe strings-import formulation which:

1. Fetches per-folder ("Get all JS strings from *.js files in a specific folder.")
2. Loads with await and a helper
3. Derives path automatically from the import.meta info.

[ufundo]

This seems great.

Small quibble: the mixin is called elements and extensions already have subfolders called settings and tests so shouldn't this subfolder be elements?

[totten]

In the MDN docs for Web Components, they point to using <template> and <slot> -- and give some examples.

The reference docs for <template> also present more nuanced attributes (like shadowrootclonable), which I don't fully understand, but it seems the component-author needs to control those attributes.

I guess... if we're going to autoload the .js and .css, then... it probably makes sense to autoload an .html file too?

• Component-author provides 1-3 files:
  • foo-bar.js
  • foo-bar.html
  • foo-bar.css
• Runtime says: "We need to render <foo-bar>, so...."
  1. Add <link> for foo-bar.css to <head> (if applicable)
  2. Add <template>s from foo-bar.html to <body> (if applicable)
  3. Run foo-bar.js (to register customElement)

(Actually... if you just had foo-bar.html, then that one file could include a mix of <style> and <template> and <script> tags... and IDEs would be fine with it... but I suppose you want to be thoughtful about how paths are evaluated for transitive-dependencies...)

[ufundo]

Suggested change

	const SUBDIR = 'element';
	const SUBDIR = 'elements';

[totten]

Yeah, it is slightly annoying how inflections vary within Civi. My first reaction was: "Right, consistency is good; fine, let's rename it". And I kind of like how ./elements/ resembles ./templates/. To implement this, it does need a slightly bigger commit:

• Prefer plural ( elements@1 + ./elements/): 8bf36ec

But I'm a sucker for this kind of thing. If we're going for consistency, then one should look the existing top-level folders and mixins:

• Folder Names: You can argue that top-levels look more plural or more singular. Really, less than half of folders (~9) indicate a free and unambiguous choice of singular vs plural. (To my eye, those lean plural.) The majority are blurry cases (~16) with abbreviations and proper-names. (To my eye, those lean singular.)
• Mixin Names: For mixins... the "unambiguous choices" and the "blurry cases" both lean toward "singular" (though plural cases do appear as well).

So... I think that singular is more consistent, in which case -- here's the commit going the other direction:

• Prefer singular (element@1 + ./element/): 1decafd

Of course, there's a lot that we inherit from upstream ecosystems -- bower_component[s]/, vendor[s]/, js/, css/, bin[s]/, etc -- and they don't point in consistent directions either...

[ufundo]

Right, I fear we will have this argument until the end of time...

My view is consistent at least: plurals are more human readable, and thereby well worth one extra character. Anything like this we add should use plurals.

[colemanw]

My vote would also be for elements which is consistent with the sibling settings and templates directories.

But whichever we do we should merge this PR and get on with our lives.

[ufundo]

Couple of ideas for translation:

clientside: use a web component for lazy fetching

• quick implementation civi-ts experiment ufundo/civicrm-core#11
• I think this is good because its relatively simple. that patch seems to work for me
• it's very safe - setting innerText protects you from any html injection (likewise setting attributes directly)
• it opens up the potential of switching languages clientside. not sure how important that is to anyone, but it's kinda neat. it could be nice for providing a "Translator's View" where you can see the src string and the translation and update as you go.
• not sure how to handle attributes (use direct translation where its needed? support some kind of prefix civi-ts-aria-described="english description"?)

serverside: have a dedicated smarty template for each component - render it serverside (including translations) and then send to the client (as an async request? by including in big JS var of rendered templates?

• good because it uses existing syntax
• gives you the power of other smarty stuff
• the power can be abused... in particular escaping remains a challenge
• needs some thought about how the rendered templates actually make their way to the client
• maybe ultimately caches better (you can cache the rendered template for each component individually)

I'm wondering whether we might want to have both available 🤔

[vingle]

Apologies if this is an obvious question, but how would a theme overwrite mixin/elements@1/example/element/shimmy-tag-a.css? In a similar way to extensions at present? e.g. riverlea/core/org.civicrm.afform_admin-ang/afGuiEditor.css?

At the moment there is a directory of 'component' css files in River (that obvs aren't web components, just thematic groups), it's been a hope that as Civi moves to Web Components, the css files for those can be managed in one place in the theme layer (rather than lots of quite specific directory names pointing to lots of css files dotted across Civi.

[ufundo]

how would a theme overwrite mixin/elements@1/example/element/shimmy-tag-a.css?

I think the idea would be this wouldn't be possible or required. Any css included in the web component itself should be extremely minimal, self-contained, and only functional not thematic. Theming would then apply on top.

[ufundo]

Here's a stab at the serverside translation approach: https://github.com/ufundo/civicrm-core/pull/12/changes

There's some crossover with the loader here - I think it shouldn't be too hard to reconcile, though I am stumbling on how "core" web components would work with this mixin loader. And possibly some architecture should go into core that can't be mixin-ed (e.g. new route?).

[vingle]

I think the idea would be this wouldn't be possible or required. Any css included in the web component itself should be extremely minimal, self-contained, and only functional not thematic. Theming would then apply on top.

How would you separate 'only functional' from 'thematic' css? This is the 189-line SCSS that Bootstrap 5 has to define radio/checkboxes: https://github.com/twbs/bootstrap/blob/main/scss/forms/_form-check.scss - that's arguably mostly 'functional' - styling is abstracted via variables. Civi's radio-button component css might be longer given requirements for specific count of columns, help text, error messaging, etc.

E.g. the gap between a radio button and its label is functional but themes will vary the size of that. There's multiple ways to do that - with margin left/right, padding, or gap if the entire thing is wrapped in flexbox or grid. If a theme has an opinion on that it might have to replace lots of 'functional' radio/checkbox css - and then the files would load twice?

I'm not saying component css files need to be over-writeable in the same way extension's css files currently are - but if they're not, then they'd either want to use css variables to support broad customisation, or themes are will have to duplicate a lot of that css.

[totten]

Random thoughts:

• In Afform, we made a big change to support multilingual by removing ts() from HTML -- instead, the HTML-loader supports selector-based translation.

  The upshot is that it's easier to view/edit/filter the HTML with basic tooling.

  For elements that use HTML templates... should we do similar selector-based translation?

• As we get deeper into the coding-conventions for WebComponents (CSS files; HTML files; theming overrides; etc), I'm feeling more conscious about the distinction between "autoloader" and "element framework". This PR conflates them a bit, because it adds an autoloader... and it also defines various conventions for folder-naming, file-naming, etc.

  An autoloader (akin to \Composer\Autoload\ClassLoader / spl_autoload_register(), but for JS / DOM / MutationObserver) should have a fairly slow cadence. (The feature-set is fairly discrete. It wouldn't need to change very much.) But the conventions for how to write a custom-element could change more (it has a longer stabilization period).

  Here is an effort to pull-out a purer implementation of the autoloader: https://github.com/civicrm/html-autoloader

[colemanw]

Here is an effort to pull-out a purer implementation of the autoloader

Nice!

[colemanw]

@totten so now we need to require https://github.com/civicrm/html-autoloader in civicm/composer.json?

[ufundo]

@totten so now we need to require https://github.com/civicrm/html-autoloader in civicm/composer.json?

I think so. Though first I think we'd need to publish a composer package? Can't we just include https://github.com/civicrm/html-autoloader/blob/main/src/HtmlAutoloader.js here @totten ?