Feat/zotero migration#3
Merged
Merged
Conversation
Add a fully local Zotero migration backend (no Translator): read zotero.sqlite
(+ storage/) and reconstruct each regular item as a Zotero-API-JSON object, then
reuse map_zotero_item + the paper-shell/catalog pipeline so ids/citekeys and
field mapping match the existing file/magic-wand import.
- services/lookup/zotero_db.rs: scan_zotero (preview counts) and migrate_zotero
(per item: folder + NOTES/highlights shell + catalog upsert). Optional PDF copy
from Zotero storage into {paper}/{id}.pdf, then liteparse PAPER.md.
- Read a private temp copy of zotero.sqlite (+ WAL) to tolerate a running Zotero.
- Dedup by arXiv id / DOI / normalized title; suffix the folder on citekey clash.
- commands/zotero.rs: zotero_scan, zotero_migrate; registered in lib.rs.
- Unit tests: item reading (excludes attachments/notes/trashed), citekey mapping,
attachment path resolution, and scan counts.
Front end for the one-click Zotero migration: - lib/zotero-migrate.ts: scanZotero / migrateZotero invoke wrappers + folder picker. - components/layout/zotero-migrate-dialog.tsx: pick the Zotero data folder, preview reference / local-PDF counts, toggle "copy PDFs into the vault", run with progress in the background-tasks panel. - App: "Migrate from Zotero" entry in the Library toolbar; refresh tree/graph/library when done. - i18n: sidebar.zoteroMigrate.* (en + zh-CN).
- backend/identifier-lookup.md: new section 16 for the local Zotero migration (zotero_scan / zotero_migrate, read path, PDF copy toggle, dedup, non-goals). - development/roadmap.md + todo.md: mark local Zotero migration done (standalone BibTeX / RIS files still go through Library import; Zotero notes / annotations / collection hierarchy still pending). - frontend/ui.md: Library toolbar "Migrate from Zotero" entry + dialog.
Migration was flat — every item landed directly under one target folder. It now
mirrors the Zotero collection tree.
- zotero_db.rs: read collections / collectionItems, build each item's collection
folder path (parent chain, sanitized) and place it at {parent}/<path>/<id> when
enabled. Collection names are also added as tags (an item can be in several);
multi-collection items pick a deterministic single path (smallest full path).
- Dialog: "Recreate Zotero collections as folders" toggle (default on);
ZoteroMigrateArgs.preserve_collections.
- i18n (en + zh-CN) + docs (identifier-lookup §16, roadmap, todo).
Deleting paper folders outside the app (e.g. in Finder/Explorer) left rows in catalog.sqlite, so the Library showed ghost papers and re-import was blocked (dedup matched the stale rows and skipped everything). - catalog: papers::prune_missing removes rows whose folder is gone from disk. - migrate_zotero prunes orphans before building the dedup set and reports the pruned count, so a re-import just works without hand-editing the SQLite catalog.
Migration always imported the whole library. The scan now returns each Zotero collection (plus an "unfiled" bucket) with its item count, and the dialog shows a checkbox list so users can import only the folders they want. - zotero_db.rs: scan returns collections; migrate accepts include_collections (0 = unfiled, absent = all) and skips items outside the selected set. - Dialog: collection checkbox list with select-all/none; ReadItem carries the raw collection ids for filtering. - i18n (en + zh-CN) + docs (identifier-lookup §16, todo).
Users with lots of Zotero notes wouldn't switch without them. Each item's child notes are now converted from HTML to Markdown (via htmd) and appended to the paper's NOTES.md on import. - zotero_db.rs: read itemNotes per paper; append_notes converts + writes; scan reports the note count; ZoteroMigrateArgs.migrate_notes toggle (default on). - Dialog: "Migrate Zotero notes" checkbox; scan preview shows the note count. - Cargo: add htmd 0.5 (pure-Rust HTML to Markdown). - i18n (en + zh-CN) + docs (identifier-lookup §16, roadmap, todo).
Re-importing skipped papers that were already in the catalog, so turning on note migration did nothing for libraries imported earlier — dedup returned before the note-writing step ever ran. migrate_one now, on a dedup hit, appends any missing Zotero notes to the existing paper's NOTES.md (idempotent by content; it never overwrites your edits) and reports them as notes_added. - Dedup maps each key to the existing paper path; append_notes is idempotent. - ZoteroMigrateResult.notes_added (TS notesAdded + pruned) exposed for a summary.
- Auto-detect the default ~/Zotero data folder on open (homeDir + join), so most users get an instant preview without browsing for it. - Show a result summary after migrating (imported / notes added / PDFs copied / stale entries cleaned / skipped / failed) instead of silently closing. - Scan preview as stat cards (papers / with PDF / notes); minor visual polish. - i18n (en + zh-CN) + docs (identifier-lookup §16).
- scan returns a per-item list (id/title/year/hasPdf/notes/collections) for a
paper-level picker; migrate accepts include_items (per-paper selection, which
takes precedence over include_collections).
- read PDF annotations (itemAnnotations: highlight + comment + page) and fold
them, with child notes, into NOTES.md via a shared idempotent appender;
migrate_annotations toggle. (highlights.md rendering is still future, so the
annotations land in NOTES.md where they are visible today.)
- stream {current,total} progress through a Tauri Channel from zotero_migrate.
- MigrateFlags groups the toggles; tests cover annotations + scan items.
…ered options
- Pick exact papers: a searchable, per-item checkbox list with a folder filter
and select-all/none; migrate sends include_items when a subset is chosen.
- Live progress bar (Tauri Channel → {current,total}) during migration.
- "Migrate PDF annotations" toggle (folded into NOTES.md for now).
- Remember the last options (copy PDFs / collections / notes / annotations /
target folder) in localStorage.
Update identifier-lookup §16 (per-item include filter, annotation text into NOTES.md, progress channel, remembered options), roadmap, and todo.
- The dialog overflowed off-screen once a library was scanned (no height cap). Cap DialogContent at max-h-[85vh] and make the body flex-1 + scroll; the header and footer stay put. - Add a "View import tutorial" link button in the footer's bottom-left; it opens IMPORT_TUTORIAL_URL via the opener plugin (falls back to window.open).
- Replace the native <select> folder filter with the themed shadcn Select; the native one rendered white-on-white (only the hovered option was legible). - Widen the dialog (sm:max-w-2xl), enlarge the search input, and give the paper list more height so the form no longer feels cramped.
The scan-count stat cards added noise; remove them (plus the now-unused Stat component and stat* strings). The scanning spinner stays.
Migrated highlights were one-per-block with a lone "— p.N" line and a `---` between each, which read as clutter. Now each highlight is a single quoted line with an inline "(p. N)" and its comment underneath, and all highlights are grouped into one block (a single leading `---`).
The description line and the "folder that contains zotero.sqlite…" hint were
redundant with the title and folder button; remove them (plus the unused i18n
keys). DialogContent stays accessible via aria-describedby={undefined}.
Deploying motif with
|
| Latest commit: |
28c1378
|
| Status: | ⚡️ Build in progress... |
There was a problem hiding this comment.
Pull request overview
Adds an end-to-end “one-click” Zotero migration flow to the desktop app: the UI can scan a local Zotero data directory and migrate selected items into the vault + catalog, optionally copying PDFs and backfilling notes/annotations, with progress reporting and updated documentation.
Changes:
- Introduces a new Zotero migration dialog + app entrypoint (Library toolbar).
- Adds Tauri commands and a Rust service to read
zotero.sqlite+storage/and migrate into the catalog (with dedup + orphan pruning). - Updates i18n strings and product docs/roadmap/todo to describe the new feature.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/lib/zotero-migrate.ts | Frontend Host API wrapper for scan + migrate commands. |
| src/components/layout/zotero-migrate-dialog.tsx | New dialog UI for scanning, selecting items, running migration, and showing results. |
| src/App.tsx | Adds a Library-toolbar button to open the migration dialog. |
| src-tauri/src/services/lookup/zotero_db.rs | New Rust implementation: read Zotero SQLite, map items, migrate into vault/catalog, stream progress. |
| src-tauri/src/services/lookup/mod.rs | Exposes zotero_db module and its types/functions. |
| src-tauri/src/commands/zotero.rs | New Tauri commands zotero_scan and zotero_migrate. |
| src-tauri/src/commands/mod.rs | Registers the new zotero command module. |
| src-tauri/src/lib.rs | Registers the new commands with the Tauri invoke handler. |
| src-tauri/src/services/catalog/papers.rs | Adds prune_missing to remove orphan catalog rows before import. |
| src-tauri/Cargo.toml | Adds htmd dependency for note HTML→Markdown conversion. |
| src-tauri/Cargo.lock | Locks new transitive deps for htmd. |
| src/i18n/locales/en/sidebar.json | Adds English strings for the migration UI. |
| src/i18n/locales/zh-CN/sidebar.json | Adds Simplified Chinese strings for the migration UI. |
| docs/frontend/ui.md | Documents the new Library-toolbar migration entrypoint and behavior. |
| docs/backend/identifier-lookup.md | Adds a new section documenting the local Zotero migration design/behavior. |
| docs/development/todo.md | Marks Zotero migration tasks completed and expands remaining follow-ups. |
| docs/development/roadmap.md | Updates roadmap to reflect Zotero migration completion. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+77
to
+83
| /** "View import tutorial" target. Replace with your hosted tutorial/docs URL. */ | ||
| const IMPORT_TUTORIAL_URL = | ||
| "https://github.com/poco-ai/motif/blob/main/docs/backend/identifier-lookup.md"; | ||
| function openTutorial() { | ||
| void openUrl(IMPORT_TUTORIAL_URL).catch(() => { | ||
| window.open(IMPORT_TUTORIAL_URL, "_blank"); | ||
| }); |
Comment on lines
+232
to
+235
| let total = items.len(); | ||
| for (idx, item) in items.into_iter().enumerate() { | ||
| progress(idx, total); | ||
| if let Some(set) = &include_items { |
Comment on lines
+63
to
+68
| const res = await invoke<ApiResult<ZoteroScan>>("zotero_scan", { | ||
| args: { zoteroDir }, | ||
| }); | ||
| if (!res.ok || !res.data) { | ||
| throw new Error(res.error?.message ?? "zotero_scan failed"); | ||
| } |
Comment on lines
+492
to
+495
| {it.hasPdf ? ( | ||
| <span className="shrink-0 rounded bg-muted px-1 text-[10px] text-muted-foreground uppercase"> | ||
| </span> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.