Skip to content

feat!: resolve catalog plugins from the @data-fair registry#28

Merged
albanm merged 30 commits into
masterfrom
feat-registry
May 27, 2026
Merged

feat!: resolve catalog plugins from the @data-fair registry#28
albanm merged 30 commits into
masterfrom
feat-registry

Conversation

@albanm
Copy link
Copy Markdown
Member

@albanm albanm commented May 27, 2026

Catalog plugins are now resolved from the central @data-fair/registry instead of a local plugins volume. The admin "install plugins" page is removed; plugin management lives in the registry.

What changes

  • API + worker: new privateRegistryUrl / secretKeys.registry config; both services call ensureArtefact to download + cache plugin tarballs on demand. The plugins volume becomes read-only and only for the v1.0.0 boot migration.
  • API surface: POST /api/plugins, DELETE /api/plugins/:id, GET /api/plugins, GET /api/plugins/:id/thumbnail, GET /api/plugins-registry are all gone. GET /api/plugins/:id now returns the registry-resolved descriptor.
  • UI: the catalogs and create-catalog pages list plugins straight from /registry/api/v1/artefacts (same domain, browser session cookie). Thumbnails come from /registry/api/v1/thumbnails/:id/data. The /admin/plugins page is deleted.
  • Migration: upgrade/1.0.0/01-publish-plugins-to-registry.ts walks the legacy <dataDir>/plugins directory, repacks each plugin into an npm-style tarball (forcing license: AGPL-3.0-only), publishes it to the registry as public: true, and best-effort uploads the in-tarball thumbnail and localized title/description. Idempotent (skips artefacts already uploaded).
  • Dev env: new registry service in docker-compose.yml, proxied under /registry/ by nginx, with a REGISTRY_PORT slot in .env.

Why

Centralizes plugin distribution and access control in the registry — operators stop shipping plugins per catalogs deployment, and the registry's per-account access rules become the single source of truth.

Regression risks (please review)

  • dataDir is now optional (defaults to null). Existing deployments that previously mounted /app/data must keep DATA_DIR set for the v1.0.0 boot to find and republish their legacy plugins; once the migration succeeds the env var can be removed. Worth calling out in the release notes.
  • Volume health check at api/src/admin/status.ts:8 silently returns OK when dataDir is unset. Cosmetic regression in /api/admin/status for ops dashboards.
  • Worker error path at worker/src/worker.ts:128-152 now catches any registry exception (not just 404/403) and marks the task error. Previously only a missing import did. A registry hiccup will permanently fail tasks until an operator retries — confirm this matches the intended behaviour.
  • Catalog creation UX: pages/catalogs/new.vue fetches /api/plugins/:id under the session account, but the catalog is created under newOwner. If a user picks a plugin they can access but their selected owner cannot, the create call 403s at the very end. UX recovers but the picker doesn't pre-filter.
  • Port fallbacks diverge: tests/support/registry.ts falls back to port 5601, while api/config/development.mjs and worker/config/development.mjs fall back to 8089, and .env uses RANDOM_NB + 33. Only the .env value works; tests run outside the worktree env will hit the wrong port silently.
  • Worker doesn't ensureDirSync(registryCacheDir) (the API does). Relies on ensureArtefact to create the cache directory. If that ever changes, the first task fails.
  • Worker debug log loses the resolved plugin version (Using plugin ${catalog.plugin} only). Minor observability regression.

Out of scope (deliberately deferred)

  • The legacy <dataDir>/plugins directory is left in place after migration for manual reconciliation. A separate v2.0.0 cleanup will remove dataDir entirely.

albanm and others added 30 commits May 21, 2026 09:11
Approved brainstorming design for replacing the local plugins volume
with @data-fair/registry: plugins resolved on demand via
@data-fair/lib-node-registry into a local cache, UI lists artefacts
from the registry API directly, and a v1.0.0 boot migration publishes
existing on-disk plugins to the registry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task-by-task plan covering dev-environment wiring, config, the shared
plugin loader, registry-backed plugin resolution, the worker switch,
the v1.0.0 boot migration, UI changes and cleanup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… volume

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace all UI plugin-list and thumbnail references that pointed at the
removed catalogs API endpoints (/api/plugins, /api/plugins/:id/thumbnail)
with direct calls to the registry (/registry/api/v1/artefacts). Add an
e2e test verifying the picker card is visible for the mock plugin.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Make getPlugin's account param required (hardening)
- Remove npm-proxy config/startup code (server.ts, config schema/defaults)
- Drop multer, semver, tmp-promise, memoizee and their @types from api deps
- Add @types/express as a direct devDep (was transitively provided by @types/multer)
- Bump root package version to 1.0.0
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Also refine the design spec: the extraction helpers move to a config-free
sibling module so they can be unit-tested without loading worker config.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These were working documents used while implementing the registry
migration; they were not meant to ship with the feature PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@data-fair/lib-node/upgrade-scripts imports every file in
upgrade/<version>/ and reads script.default.description — a helper
module without a default export crashed the worker on boot with
"Cannot read properties of undefined (reading 'description')".

Move plugin-metadata.ts up one level (the framework filters
top-level files out of upgrade/) and update the two import paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev compose pointed at the feat-npm-noarch branch image while
that work was in flight; main now ships the same npm-upload + flat
artefact path the catalogs API and the v1.0.0 upgrade script rely on.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The workspace dev scripts wrapped nodemon in a pipeline
(nodemon … | tee ../dev/logs/dev-*.log), entangling nodemon's
restart cycle with the tee process and causing EADDRINUSE when the
new node child raced the old one's port release.

Move the tee one level up to the root dev-{api,worker,ui} scripts —
nodemon (and vite) now run as a direct child of npm with no sibling
pipe. Matches the processings repo's working pattern. Also align
ignore lists in nodemon.json with processings (../data/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- api: static import for registryCacheDir in the test-env router
  (was a dynamic await import for no reason).
- tests: drop the installMockPlugin pass-through in tests/support/axios.ts
  and have permissions.api.spec.ts import publishMockPlugin directly.
- tests: drop the dead form.append('architecture', …) from publishMockPlugin
  (the upgrade script doesn't send it either; the noarch registry ignores it).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The release process owns the version bump; the upgrade/<version>/ folder
must match the currently released version so the upgrade-scripts lib
picks it up on instances coming from that version.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@albanm albanm merged commit f4d1598 into master May 27, 2026
4 checks passed
@albanm albanm deleted the feat-registry branch May 27, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant