experimental: relay based posting approach#873
Merged
Conversation
Adds a standalone, dependency-free prototype of a proposed "Relay" posting framework (job tree + staged pipeline + unified NDJSON trace) under prototypes/relay, runnable via `node --experimental-strip-types`. Includes design rationale and a concrete plan for landing it in the real NestJS + Drizzle + Electron app (schemas, entities/repos, engine port, UI wiring, migrations, and a 6-PR rollout). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR #1 of the Relay posting framework rollout. Adds the persistence layer for PostUnit) plus a persisted rate-limit window, with no behavior the existing post-manager remains in use.change Types: - NodeStatus, UnitKind, PostErrorKind enums - IPostJob / IPostTask / IPostUnit, Dependency (all|any|count), ITaskError, IPostRateWindow, and the JobTreeNode UI projection - PostJobDto / PostTaskDto / PostUnitDto Database: - post-job, post-task, post-unit, post-rate-window schemas (JSON columns for dependency/error/fileIds/accountSnapshot; indexes; cascade + set-null FKs) - entities with hydration, repositories with findActive / findBySubmission / findByJob / findByTask / findByKey - drizzle migration 0008 - integration spec covering tree hydration, JSON round-trip, terminal filtering, rate-window lookup, and cascade deletes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR #2 of the Relay rollout. Ports the prototype engine into the NestJS app as injectable, unit-tested services. Gated by a new `useRelayEngine` settings flag (default the legacy post-manager remains the active posting path,off) so there is no behavior change. Engine (apps/client-server/src/app/post/engine): - model: in-memory job tree (RelayJob/RelayTask/RelayUnit), node state machine, computeJobStatus, and evaluateDependency (all/any/count + blocked detection) - errors: StageError + typed-kind retry policy (decideRetry) - transform: declarative plan -> verify loop + content-addressed cache, with a swappable Encoder seam; SharpEncoder wraps the existing SharpInstanceManager - rate-limiter: scope-aware rateKey + RateStore (Memory + SqliteRateStore over the post-rate-window table) so windows persist and are shared across jobs - tracer: per-job NDJSON + JobTreeNode projection + POST_STATE_DELTA WS deltas - websites: RelayWebsite contract + WebsiteInstanceAdapter reading decoratedProps - pipeline: staged Resolve->Settle pass with injectable seams (testable) - scheduler: queue, concurrency, dependency gating, WAITING, retry orchestration Supporting changes: - types: useRelayEngine settings flag; website rateLimitScope + sourceDependencyMode decorator props - socket-events: POST_STATE_DELTA event; post.events delta type - post.module: register RelayTracer, RateLimiter, SharpEncoder providers Tests: 18 specs (model/transform/rate-limiter unit + a full pipeline+scheduler integration covering batching, propagation, retry, rate-limit wait, resume, any-mode deps, and skips). typecheck + lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR #3 of the Relay rollout. Connects the engine seams to PostyBirb's real services and adds a dry-run preview. Still behind the `useRelayEngine` flag; the legacy post-manager remains the active path. - transform: Encoder now returns EncodeResult{bytes, buffer?} and RelaySource files/TransformedFiles carry buffers, so the converged verify result holds the exact bytes that will be posted (not just a size estimate) - sharp-encoder: performs a real resizeForPost each verify iteration via the worker pool, returning the encoded buffer + byte length - pipeline-deps: production PipelineDeps loadSubmission maps awiring Submission (files + buffers + per-account dimension overrides) to the engine model and resolves website instances/options; buildPostData uses PostParsersService (+ injected upstream source URLs); validate uses ValidationService; dispatchFile/Message post the resized PostingFiles - preview.service: dry-run that runs plan + transform per website with no dispatch and no persistence; exposed at GET /post/:id/preview - post.module: register RelayPipelineDeps + RelayPreviewService Tests: preview.service.spec (resize results, exclusions, error capture) plus the updated transform spec; 21 engine specs pass. typecheck + lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…utover PR #4 of the Relay rollout. Persists the job tree, recovers interrupted jobs on boot, and wires the queue cutover behind the useRelayEngine flag (default the legacy post-manager remains the active path).off - model/pipeline: job-scope task/unit ids (<jobId>:t:<site>:<acct> / <taskId>:b<n>) so they are unique DB primary keys across jobs/attempts - persistence: RelayPersistence maps RelayJob to post-job/post-task/post-unit (create + idempotent save/saveTask upserts) and loadActive() reconstructs non-terminal job trees for recovery - scheduler: optional onTaskChanged/onJobChanged hooks (persist every transition) and adopt() to register recovered/pre-planned jobs - post-manager.service: RelayPostManager orchestrates enqueue, plan, persist, run, serialized drain, terminal archive/notify, cancel, and OnModuleInit crash recovery (loadActive + resume after registry init); guards against duplicate jobs per submission - post-queue.service: executeRelay() when the flag is on, thecutover post-queue table stays the source of truth and the RelayPostManager owns running (start / dequeue-on-terminal / leave-running); legacy path untouched when off Tests: persistence integration (create/reload, transition persistence, RUNNING recovery) plus existing engine specs = 24 pass. typecheck + lint clean; existing post-queue spec still green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…store PR #5 of the Relay rollout. Gives the UI a single live source of truth for posting state, replacing the bespoke wait-state assembly. - post-manager: getActiveTrees() projects all in-flight jobs to JobTreeNode - post.controller: GET /post/jobs/active returns the snapshot - ui post.api: getActiveJobs() - ui posting-state-store: zustand store keyed by jobId, seeded from the snapshot and kept live by POST_STATE_DELTA deltas (job/task/unit merged into the owning subtree); terminal jobs are pruned. Exposes useActiveJobs and usePostingStateActions selectors. Still behind the useRelayEngine flag; deltas only flow when the engine runs. typecheck + lint clean; 24 engine specs pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR #6 of the Relay rollout. Surfaces persisted job-tree history so the UI can render past and in-flight posts from one model. - persistence: loadBySubmission() returns all job trees for a submission (newest first), terminal ones included - post-manager: getHistory() projects those trees to JobTreeNode, overlaying any in-flight job from memory so it reflects live state - post.controller: GET /post/:id/jobs - ui post.api: getJobHistory(submissionId) Tests: persistence loadBySubmission spec (terminal job retained in history); 25 engine specs pass. typecheck + lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Final piece of the Relay rollout UI. Adds a recursive JobTreeView that renders unit) with status icon/color, progress bar, per-node rate-limit countdown, source-url links, and error tooltips. The PostingActivityPanel now seeds the posting-state store from GET /post/jobs/active on mount and renders any active relay job trees above the legacy cards. Relay trees only exist when the engine is running, so the panel is unchanged when the useRelayEngine flag is off. typecheck + lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Makes the relay file path reproduce the legacy pipeline exactly and isolates concurrent jobs. - file-processor: RelayFileProcessor reuses PostFileResizerService (sharp worker pool) + FileConverterService, porting resizeOrModifyFile, getResizeParameters (website calculateImageResize + per-account dimension overrides + static/dynamic size limits), alt-file text fallback, thumbnail generation, source-url/alt-text metadata, and verifyPostingFiles mime check - pipeline: Transform stage now delegates to deps.processBatch -> PostingFile[] (real convert/resize/thumbnail), dispatch posts those; context lookups take jobId - pipeline-deps: per-job context keyed by jobId (prepare/release), processBatch via the processor; removes the estimate-only encoder/cache from the real path - scheduler: split createJob/plan so the manager prepares context between them - post-manager: prepare(job) before planning, release(jobId) on completion, enqueue accepts priority/scheduledFor - preview: self-contained dry run using the real processor (no dispatch) Tests: scheduler integration harness updated to the processBatch seam; 22 engine specs pass. typecheck + lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Removes the legacy posting implementation from the backend now that the Relay engine is production-faithful. Relay is the only executor. Removed: - post-manager-v2 (BasePostManager, File/Message managers, PostManagerRegistry) - post-record-factory (PostRecordFactory + local PostEventRepository) - their specs Rewired: - post-queue.service: lean, relay-only. The post-queue table records what to post; RelayPostManager owns running. Each cycle starts a Relay job for queued submissions, dequeues finished ones, leaves running ones alone. Keeps pause/resume, scheduled-submission cron (incl. recurring next-run), and startup grace. No more PostRecord creation, crash-recovery, or stuck-record handling here (RelayPostManager owns recovery). - post-manager.controller: cancel/:id now just delegates to queue dequeue (which cancels the Relay job) - post.controller: drop legacy wait-states endpoint - post.module: drop legacy providers/exports; export RelayPostManager - tracer: cap the in-memory entry buffer (disk NDJSON remains the full record) The PostRecord/PostEvent schema + repositories remain as the historical read model the existing submission UI still reads; Relay history is additionally available via GET /post/:id/jobs. Tests: post-queue spec rewritten against RelayPostManager; 34 post specs pass. typecheck, lint, and full build green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrates the UI off the now-empty PostRecord data (the relay engine no longer writes it) onto the job-tree model. - types/tracer: JobTreeNode carries submissionId (set on job projection) so the UI can correlate a job tree to its submission - posting-state-store: add submission-keyed selectors useActivePostingSubmissionIds, useIsSubmissionPosting, useSubmissionActiveJob - isPosting consumers: submission-edit-card-actions + submission-picker now read live posting state from the relay store instead of submission.isPosting - posting-activity-panel: rewritten active posts render as live joblean trees; queued = queued submissions not already posting (no double-display). Drops the legacy event-derived ActivePostCard/helpers - post-history-content: fetches GET /post/:id/jobs and renders job trees (newest first) with the live in-flight job overlaid from the store; re-fetches when a post starts/completes Per-account "previously posted" badges in the edit-card account selection and post-confirm modal still read PostRecord events and degrade to empty for relay posts (PostRecord read-model retained); a follow-up can migrate those. typecheck, lint, build, and 22 engine specs green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Finishes the UI migration off PostRecord and removes the now-dead feature flag. - JobTreeNode/tracer: task nodes carry accountId - history-utils: add derives a per-account status mapgetAccountStatusFromJob from a job tree's task statuses (success/failed/running/rate-limited/waiting) - selected-accounts-forms: per-account posting badges now come from the live Relay job (useSubmissionActiveJob) instead of submission post records - post-confirm-modal: drop the PostRecord-based resume-skip prediction (the relay queue path always enqueues fresh and handles resume internally); the account pill list now simply lists the configured websites - settings: remove the dead useRelayEngine flag (relay is the only executor) typecheck, lint, build, and 22 engine specs green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tear out the legacy PostRecord/PostEvent data model now that the relay engine is the sole posting executor. Drop the schemas, entities, repos, DTOs, the PostRecordState enum, PostService, and dead UI affordances (post-record card, wait-state store, resume-mode prompts). Migration 0009 drops the post-record and post-event tables and rebuilds post-queue without the postRecordId FK. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The relay authenticate stage was a placeholder and dispatch never ensured login, so an expired session would fail instead of re-authenticating. Add an ensureLoggedIn on the website adapter and an authenticate seam on PipelineDeps invoked at the authenticate stage, restoring legacy parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a Preview action to the submission edit card that runs the existing relay dry-run endpoint and renders, per account, whether the website is supported and how each file would be converted/resized (before -> after dimensions, bytes, mime). Shared PreviewResult types move to @postybirb/types and back a new post.api.preview client method. Adds a RelayPreviewService unit spec. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The relay data layer and the legacy-table teardown were split across two migrations (0008 create + 0009 drop post-record/post-event, rebuild post-queue). Since the relay tables are new on this unreleased branch, regenerate them as one migration via drizzle-kit so a fresh install gets the correct schema (including post-job.version) in a single step. Local dev databases that already applied the old 0008/0009 should be recreated; no shipped data is affected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR #834 split getDatabase into test/production branches and dropped the migrate() call from the production path, so a fresh desktop database was opened but never migrated (no such table: settings/notification/...). Construct the production better-sqlite3 connection explicitly, enable foreign-key enforcement (matching the test environment and relay's cascade deletes), and run migrations before returning the instance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a per-submission debug log download (GET /post/:id/logs) that streams the concatenated NDJSON tracer files for every persisted job (falling back to the in-memory ring buffer when no file exists yet). Wire it through to the posting history drawer alongside per-job JSON view/copy/save controls and replace the in-app anchor in JobTreeView with the system-browser ExternalLink + clipboard affordance the legacy post-record card used to provide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wrap each job in the history panel's copy/save payload with the app version, remote-mode flag, and submission type/flags so the JSON a user shares for debugging is self-contained. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve task nodes through the account store so the history page shows "<Website> - <Account name>" instead of the raw account id, falling back to the engine label when the account is unknown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The standalone prototype under prototypes/relay/ has been fully ported into apps/client-server/src/app/post/engine/ and is no longer referenced by any project. Drop the directory along with its design and planning docs to keep the repo focused on the shipped engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cancelling a submission previously only flipped a boolean the scheduler checked between iterations, so a job parked on a long rate-limit window (up to several minutes) or a retry backoff would make the user wait it out before the cancel took effect. Add cancellation listeners to CancellableToken and race the scheduler's sleeps against them so a cancel returns promptly. On a cancelled exit, mark any tasks/units still parked in WAITING/QUEUED as CANCELLED so the job settles to a terminal state instead of being recomputed as WAITING. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two failure modes could leave a non-terminal post_job row in the DB that the user could not clear via the cancel UI: 1. Crash recovery iterated active jobs without per-job error isolation, so one bad job (submission deleted while the app was closed, missing website implementation, etc.) would abort the loop and leave every subsequent active job unadopted. Those rows then sat in QUEUED / RUNNING / WAITING forever because cancel() only worked when the job was live in the scheduler. 2. cancel() short-circuited to a no-op when no live job was registered, giving the user no escape hatch for a row that recovery dropped. Wrap each job's recovery in its own try/catch and force-mark a poison job terminal-FAILED via persistence.failJob(). Make cancel() fall back to persistence.cancelNonTerminalForSubmission(), which sweeps every non-terminal post_job/post_task/post_unit row for the submission to CANCELLED. Together these guarantee that a stuck row can always be cleared and that one bad row cannot strand the rest of the active set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address five validated reliability findings: 1. persist-fail no longer double-posts: separate the pipeline pass from its success persist and route all task writes through a durable retrying persist that never throws, so a DB error cannot flip a SUCCEEDED unit back to a re-postable state. 2. classify network/IO errors as TRANSIENT in all stages (authenticate, parse, validate, transform), not only dispatch, so blips retry instead of failing terminally. 3. recovery retries website-registry init and defers to an enqueue reconcile that adopts/resumes (or force-fails) an untracked non-terminal job instead of creating a duplicate post_job row. 4. add a cumulative rate-limit wait ceiling so a task on a busy shared bucket cannot be starved indefinitely. 5. arm a scheduled-run timer in enqueue so future scheduledFor jobs run when due even with no other queue activity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prevent unbounded growth in the posting engine over a long-running process: - scheduler: evict terminal jobs from the live `jobs` map (previously never deleted) into a bounded recent-completions LRU; manager calls scheduler.forget() after archive/release. History is served from the DB, so dropping finished trees from memory is safe. - transform: bound TransformCache (which can retain encoded image buffers) with an LRU cap instead of an unbounded Map. - tracer: replace the single global ring buffer with bounded per-job buffers (per-job + job-count caps) so a busy job can no longer evict another job's ledger/history. - tracer: add pruneOldLogs() retention for per-job NDJSON files, run in the background on startup, so trace logs don't accumulate forever. Adds regression tests for job eviction, transform cache LRU, per-job trace isolation/eviction, and on-disk log pruning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The plan/encode/verify/cache transform code in engine/transform.ts was a prototype superseded by RelayFileProcessor + PostFileResizerService and was only exercised by its own spec. Remove it along with SharpEncoder (an orphaned provider injected nowhere) and the unused canTransition state-machine stub. - transform.ts: trim to the live RelaySourceFile type (drop buildTransformPlan, planCacheKey, runTransform, TransformCache, SimulatedEncoder/simulatedEncoder, Encoder/EncodeRequest/EncodeResult, TransformPlan/TransformedFile/VerifyResult, WebsiteFileConstraints) - delete sharp-encoder.ts + transform.spec.ts - post.module.ts: drop SharpEncoder provider; engine/index.ts: drop barrel export - websites.ts: drop unused fileConstraints field/getter - model.ts: drop unused canTransition + ALLOWED_TRANSITIONS - scheduler.integration.spec.ts: drop stale fileConstraints mock field Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…hub.com/mvdicarlo/postybirb into users/mvdicarlo/relay-posting-framework
mvdicarlo
marked this pull request as ready for review
July 21, 2026 11:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.