feat!: resolve catalog plugins from the @data-fair registry#28
Merged
Conversation
Approved brainstorming design for replacing the local plugins volume with @data-fair/registry: plugins resolved on demand via @data-fair/lib-node-registry into a local cache, UI lists artefacts from the registry API directly, and a v1.0.0 boot migration publishes existing on-disk plugins to the registry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task-by-task plan covering dev-environment wiring, config, the shared plugin loader, registry-backed plugin resolution, the worker switch, the v1.0.0 boot migration, UI changes and cleanup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… volume Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace all UI plugin-list and thumbnail references that pointed at the removed catalogs API endpoints (/api/plugins, /api/plugins/:id/thumbnail) with direct calls to the registry (/registry/api/v1/artefacts). Add an e2e test verifying the picker card is visible for the mock plugin. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Make getPlugin's account param required (hardening) - Remove npm-proxy config/startup code (server.ts, config schema/defaults) - Drop multer, semver, tmp-promise, memoizee and their @types from api deps - Add @types/express as a direct devDep (was transitively provided by @types/multer) - Bump root package version to 1.0.0
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Also refine the design spec: the extraction helpers move to a config-free sibling module so they can be unit-tested without loading worker config. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These were working documents used while implementing the registry migration; they were not meant to ship with the feature PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@data-fair/lib-node/upgrade-scripts imports every file in upgrade/<version>/ and reads script.default.description — a helper module without a default export crashed the worker on boot with "Cannot read properties of undefined (reading 'description')". Move plugin-metadata.ts up one level (the framework filters top-level files out of upgrade/) and update the two import paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev compose pointed at the feat-npm-noarch branch image while that work was in flight; main now ships the same npm-upload + flat artefact path the catalogs API and the v1.0.0 upgrade script rely on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The workspace dev scripts wrapped nodemon in a pipeline
(nodemon … | tee ../dev/logs/dev-*.log), entangling nodemon's
restart cycle with the tee process and causing EADDRINUSE when the
new node child raced the old one's port release.
Move the tee one level up to the root dev-{api,worker,ui} scripts —
nodemon (and vite) now run as a direct child of npm with no sibling
pipe. Matches the processings repo's working pattern. Also align
ignore lists in nodemon.json with processings (../data/).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- api: static import for registryCacheDir in the test-env router
(was a dynamic await import for no reason).
- tests: drop the installMockPlugin pass-through in tests/support/axios.ts
and have permissions.api.spec.ts import publishMockPlugin directly.
- tests: drop the dead form.append('architecture', …) from publishMockPlugin
(the upgrade script doesn't send it either; the noarch registry ignores it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The release process owns the version bump; the upgrade/<version>/ folder must match the currently released version so the upgrade-scripts lib picks it up on instances coming from that version. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Catalog plugins are now resolved from the central
@data-fair/registryinstead of a local plugins volume. The admin "install plugins" page is removed; plugin management lives in the registry.What changes
privateRegistryUrl/secretKeys.registryconfig; both services callensureArtefactto download + cache plugin tarballs on demand. The plugins volume becomes read-only and only for the v1.0.0 boot migration.POST /api/plugins,DELETE /api/plugins/:id,GET /api/plugins,GET /api/plugins/:id/thumbnail,GET /api/plugins-registryare all gone.GET /api/plugins/:idnow returns the registry-resolved descriptor./registry/api/v1/artefacts(same domain, browser session cookie). Thumbnails come from/registry/api/v1/thumbnails/:id/data. The/admin/pluginspage is deleted.upgrade/1.0.0/01-publish-plugins-to-registry.tswalks the legacy<dataDir>/pluginsdirectory, repacks each plugin into an npm-style tarball (forcinglicense: AGPL-3.0-only), publishes it to the registry aspublic: true, and best-effort uploads the in-tarball thumbnail and localized title/description. Idempotent (skips artefacts already uploaded).registryservice indocker-compose.yml, proxied under/registry/by nginx, with aREGISTRY_PORTslot in.env.Why
Centralizes plugin distribution and access control in the registry — operators stop shipping plugins per catalogs deployment, and the registry's per-account access rules become the single source of truth.
Regression risks (please review)
dataDiris now optional (defaults tonull). Existing deployments that previously mounted/app/datamust keepDATA_DIRset for the v1.0.0 boot to find and republish their legacy plugins; once the migration succeeds the env var can be removed. Worth calling out in the release notes.api/src/admin/status.ts:8silently returns OK whendataDiris unset. Cosmetic regression in/api/admin/statusfor ops dashboards.worker/src/worker.ts:128-152now catches any registry exception (not just 404/403) and marks the taskerror. Previously only a missing import did. A registry hiccup will permanently fail tasks until an operator retries — confirm this matches the intended behaviour.pages/catalogs/new.vuefetches/api/plugins/:idunder the session account, but the catalog is created undernewOwner. If a user picks a plugin they can access but their selected owner cannot, the create call 403s at the very end. UX recovers but the picker doesn't pre-filter.tests/support/registry.tsfalls back to port5601, whileapi/config/development.mjsandworker/config/development.mjsfall back to8089, and.envusesRANDOM_NB + 33. Only the.envvalue works; tests run outside the worktree env will hit the wrong port silently.ensureDirSync(registryCacheDir)(the API does). Relies onensureArtefactto create the cache directory. If that ever changes, the first task fails.Using plugin ${catalog.plugin}only). Minor observability regression.Out of scope (deliberately deferred)
<dataDir>/pluginsdirectory is left in place after migration for manual reconciliation. A separate v2.0.0 cleanup will removedataDirentirely.