Outdated frontend dependencies, considering contributing

[fidergo-stephane-gourichon]

Hi,

Testing otterwiki, I and a few friends are considering adopting it. We noticed the frontend dependencies are fairly old and that upgrading them means manually replacing files. Here's what we found:

• CodeMirror 5.65.15 (latest: 6.0.2)
• Font Awesome Free 5.15.4 (latest: 7.2.0)
• MathJax 3.2.0 (latest: 4.1.1)
• Mermaid 11.6.0 (latest: 11.14.0)
• htmx 2.0.6 (latest: 2.0.8)
• Sortable.js 1.15.3 (latest: 1.15.7)

We're considering helping, e.g. by upgrading the packages and/or making future upgrades easier, preferably in a pythonic way (although unsure how applicable this is to non-Python packages).

Anyway, before proposing anything specific I'd rather ask first: you know best what fits how you maintain the project. What would be your preferences?

[redimp]

Hey @fidergo-stephane-gourichon, thank you for reaching out and your interest in An Otter Wiki.

About dependencies, I have an (until now) unwritten strategy: If it isn't broken and doesn't bring features I need, I keep the versions stable. Also, some upgrades come with a lot of work but not much gain. Let me go into the details:

• CodeMirror: I worked on moving to CodeMirror 6 for some time and gave up eventually. The code (not rebased in a long time) lives in feature/cm6-proof-of-concept. Also check the discussion in Upgrade to CodeMirror 6, make the editor configureable (was: Option to switch text editor) #105. Please go ahead and take a look—you will find mobile users very happy with it. Checklist:

  • Theme working decent, including light and dark mode.
  • Syntax Highlighting supported, including code blocks (check otterwiki.com/Examples/Syntax%20Highlighting).
  • All toolbar functionality supported (e.g., modifying markdown tables).
  • Right Sidebar tooling supported (Attachments and Wikilinks).

• Font Awesome: I didn't miss anything from Version 5, and I wasn't keen on some of the alternatives in Version 6. Please go ahead and try a mockup for Version 7; if it looks good, I'm easily convinced.

• MathJax & Mermaid: Go ahead. Please test and verify manually that everything works.

• htmx & sortable: Go ahead. These should be straightforward and easier to test.

Since we are living in this time now, I have to add this disclaimer: Please don't submit slob. Keep in mind that every PR takes probably more time for me to test than most tooling to write. Thank you!

[fidergo-stephane-gourichon]

Thanks a lot for your reactivity!

Prepared commits. Done some tests.

• Font Awesome: not sure what to test. Dev tools show fonts loaded.
• MathJax: I copy from https://otterwiki.com/Examples/MathJax/source and it renders the same in my copy, also I change things and the changes are reflected
• Mermaid: I copy from https://otterwiki.com/Examples/Mermaid/source and it renders the same in my copy, also I change things in each diagrams and the changes are reflected, also if I break the syntax an error appears instead of the diagram and the if I fix the diagram appears again
• htmx: I guess the thing would not work at all if broken.
• sortable: given it is use in sidebar_preferences.html and I have the same result as on https://demo.otterwiki.com/-/admin/sidebar_preferences I guess it's okay?
• codemirror: I can add code blocks in the editor and they are nicely colored during the edit. E.g.:

def foo():
    pass

That makes 6 commits. Should I do one PR for each, or one for the whole set?

[redimp]

Please split it in four PRs:

• (1) CodeMirror (most complicated to check)
• (2) Font Awesome (check based on taste, extra hard)
• (3) mermaid and mathjax (needs testing user examples)
• (4) htmx and sortable, should be straight forward

Will review over the next days. Please bring patience.

[redimp]

Please compare your CodeMirror PR to #464, that @onovy just submitted.

. o O ( What's going on? Is it CodeMirror Sunday while I'm at the zoo ? )

[fidergo-stephane-gourichon]

Oh sorry I forgot to clarify: I have taken the easy route, upgrade to a recent compatible version of CodeMirror 5, not take the hard path to CodeMirror 6.

[fidergo-stephane-gourichon]

Regarding CodeMirror, @onovy seems to have made a much more comprehensive and interesting work.
My changes work for me, but I am a little uncertain how "clean" they are. I have not submitted a PR for that part.

[fidergo-stephane-gourichon]

Hi again.

I'm uncertain about whether the PR's that I opened #465 #466 #467 are really okay, and I didn't even submit my changes about CodeMirror.

I'm more used to dependency handling where a project's repo only commits a reference about a dependency, like it's done in 3869aa6 .

--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,7 +20,7 @@ dependencies = [
     "cython==3.0.11",
     "mistune==3.2.0",
     "pygments==2.20.0",
-    "Pillow==12.1.1",
+    "Pillow==12.2.0",
     "PyYAML==6.0.2",
     "unidiff==0.7.5",
     "beautifulsoup4==4.12.3",

On the contrary, committing transpiled and minified files looks not very maintainer-friendly to me. I did it to follow the existing pattern yet I don't even know if it's a preference from you or you're willing to do differently.

Anyway, before proposing anything specific I'd rather ask first: you know best what fits how you maintain the project. What would be your preferences?

While I'm not particularly fond of node/npm, especially for a Python-based project, committing minified third-party JS makes dependency tracking and review harder. npm does not cause the dependency hell, the libraries do. #464 looks like a step in the right direction by introducing a package.json, and we could get more of the intended benefit if this also became an opportunity to stop committing minified files. That would make dependencies more explicit, updates easier to review, and builds more transparent. It would also allow tools like npm audit to surface known security issues, including in transitive dependencies.

Would you consider having at least some dependencies handled via a reference file? We could contribue migrating to such a model. Then, instead of commits containing minified files that are impossible to actually review, they would look like this:

index 9213651..8856a4a 100644
@@ -5,7 +5,7 @@
     "vendor": "node scripts/vendor-copy.js"
   },
   "dependencies": {
-    "@fortawesome/fontawesome-free": "5.15.4",
+    "@fortawesome/fontawesome-free": "^7.2.0",
     "codemirror": "5.65.15",
     "htmx.org": "^2.0.8",
     "mathjax": "3.2.0",

Feel totally free about this. One of my friend is more of a front-end developer than me and may contribute with a bit more perspective.

[redimp]

Hey @fidergo-stephane-gourichon,

On the contrary, committing transpiled and minified files looks not very maintainer-friendly to me. I did it to follow the existing pattern yet I don't even know if it's a preference from you or you're willing to do differently.

I had the same thought. With committed 3rd party (minified) js I basically have to load the same official versions and compare them to make sure that nothing weird is going on. In the time where LLMs can make changes easily, supply chain attacks are around every corner, I have to be extra careful. Changes are not necessary harmful but can just be broken.

I always wanted to circumvent to have a node/npm (or nowadays bun?) build system on top, including the risks coming with it, so I sticked to handpicking the releases myself.

Have to think about this.

[onovy]

Good news everyone :)

605fec8

(because for CodeMirror 6+ is it must have).

Minified CM6 is still in repo, but it can be rebuild with make cm6 anytime.

[redimp]

MathJax, Mermaid, htmx, Sortable.js dependencies upgraded in v2.20.3.

With #467 being in review and CM-6 on the way in #464 I consider this issue resolved.