feat(import): conversation and memory-export importers - #694
Conversation
people arrive with years of notes already written and a fresh kb has nothing to say to them. `vouch import obsidian <vault>` and its four siblings file one PENDING page proposal per note, each cited to a source registered from the note's own bytes. that last part is the whole point. the source content is the note verbatim, so a claim extracted from it quotes real offsets and its receipt verifies — imported knowledge is citable, not paraphrased into an embedding. it is the advantage the issue identifies and the reason this is worth building rather than documenting "convert to markdown and use the inbox". five loaders behind one normalised `Note`: - obsidian and plain markdown folders share a loader — obsidian's on-disk format *is* a folder of markdown with frontmatter and wikilinks; the only difference is `.obsidian/`, which the walk skips either way. - joplin, from a `.jex` tar or the folder it unpacks to, parsing the trailing `key: value` metadata block and skipping non-note `type_` records (importing a folder record as a note is the classic mistake with that format). - apple notes, html or txt, with a stdlib HTMLParser rather than a new dependency for one importer. - google keep from a takeout folder or zip, including checklists and skipping trashed notes. re-running is idempotent on a stable per-note identity derived from the origin's own identifier — the vault-relative path, or joplin's/keep's own id where the format carries one. unchanged notes are no-ops, changed ones refresh their pending proposal in place, decided ones stay decided. with `--limit` that makes a ten-thousand-note vault importable in resumable slices. wikilinks (and joplin `:/id` links) become `references` relation proposals in a second pass, after every source is registered, so a link is judged against the whole imported set. unresolvable targets are dropped — a vault is full of links to notes nobody wrote, and a dangling edge is not knowledge. claims go through the existing density-selection knob and are off by default: `--max-claims 0` means pages only, so a first command cannot turn a big vault into ten thousand pending claims. nothing calls approve(). an import is a proposal firehose, not a write.
the diff-coverage gate wants 100% of changed python, and the uncovered lines were the skip-and-carry-on branches — which are the ones a vault importer lives or dies on, since a vault is other people's files and one malformed note must not fail an import of ten thousand. added: body clipping, the two frontmatter non-fence cases, comma-string tags, the per-note wikilink ceiling, machinery/dotfile skipping, a file that vanishes mid-walk, the byte ceilings for folder/jex/zip, both timestamp helpers degrading on junk, joplin footers that are prose rather than metadata and its jex non-note members, the html parser blowing up, apple-notes and keep loaders on a file/empty export/missing path, keep skipping empty, trashed and undecodable notes, a wikilink written with its extension, and the link pass leaving an unchanged target alone. plus one cli test per subcommand — the group is the surface the issue asks for, and a subcommand wired to the wrong kind would otherwise be invisible — and the human-readable output path. the one branch left is tarfile's `extractfile() is None` for a member `isfile()` already vouched for; marked no-cover with the reason rather than faked.
the vault walk guarded only the explicit `path.stat()`, but `is_file()` reaches the filesystem through `Path.stat` on python 3.11/3.12 and through `os.stat` on 3.13+. so a file that disappears mid-walk was skipped on one version and raised on another — which is exactly what the new test caught: green locally on 3.14, red on all three ci pythons. one guard around every stat-backed call in the loop instead.
someone arriving with existing agent history starts with an empty kb and
re-teaches everything by hand. `chatgpt_import` covers one vendor and the
note-vault importers cover files; the gap is every other chat export, and
the memory dumps prior memory tools emit.
`vouch import chat-json <export>` normalises the shapes that matter into
the `Conversation`/`Exchange` pair the chatgpt importer already uses:
openai's branching `mapping` tree (recognised and delegated rather than
parsed a second way), claude.ai's `chat_messages` with `sender` and typed
blocks, the generic `messages: [{role, content}]`, and a bare message
list with no conversation wrapper. from a file, a .jsonl, or a .zip. one
pending page per conversation, cited to a per-conversation source, with
the same stable session id that makes re-imports idempotent.
`vouch import memory-export <dump>` reads a json array, an object of
records, a wrapper key, jsonl, or one memory per line, and files each
memory as a claim quoting its own source verbatim — so the receipt
verifies by construction and an imported fact is citable, not asserted.
`markdown-vault` is the third format the issue names; it is the
note-vault importer, aliased rather than reimplemented.
two guards, both of which the issue asks for and both of which matter
more than the readers do:
`--max-proposals` caps a whole run, charges pages and claims to the same
budget, and reports `capped` so an operator can see the import was
truncated rather than finished. rerunning continues where it left off,
which is what makes a ten-year history importable at all.
dedup drops a candidate an approved claim or a pending proposal already
covers. lexical first: a base install has no `[embeddings]` extra, and
dedup that silently stops working is exactly how an unattended import
floods a review queue. the vouchdev#147 embedding hits fold in on top when the
extra is present, and both halves are pinned by tests.
nothing here calls approve(). everything lands pending.
github-actions
Bot
added
docs
|
the two guards are correctly identified as mattering more than the readers, and i would go further — they are the only reason a ten-year history is importable at all rather than a single flood into the review queue.
on dedup, doing lexical first and folding the #147 embedding hits in on top is the right default and the reasoning is exactly right — a base install has no delegating openai's branching practical note on merge order: this is stacked on #691 and the diff currently carries all of one question: |
|
closing on mergeability and stacking — both are fixable, neither is a judgement on the work.
sequence when you come back: rebase #691 onto the substance i want kept, unchanged. the two guards are correctly identified as mattering more than the readers. and the dedup work is the part i would protect hardest: lexical first with the #147 embedding hits folded in on top, and a test that forces the import to fail so the base-install path is the one under test. CI installs one thing to reconsider: |
2 participants
closes #431
what
chat-jsonnormalises the shapes that actually turn up into the sameConversation/Exchangepair the chatgpt importer already uses:mappingtree — recognised and delegated tochatgpt_import, not parsed a second waychat_messages, withsenderand typed content blocksmessages: [{role, content}]almost everything else emitsfrom a plain file, a
.jsonl, or a.zip. One PENDING page per conversation,cited to a per-conversation source, on the same stable session id that makes
re-import idempotent — unchanged is a no-op, grown refreshes in place, decided
stays decided.
memory-exportreads a JSON array, an object of records, amemories/facts/itemswrapper key, JSONL, or one memory per line, and files eachmemory as a claim quoting its own source verbatim. The receipt verifies by
construction, so an imported fact is citable rather than asserted — which is
the thing that makes importing prior-tool memory into vouch worth doing rather
than just copying text around.
the two guards, which matter more than the readers
--max-proposalscaps a whole run. Pages and claims charge the samebudget, and the report carries
cappedso an operator can see the import wastruncated rather than finished. Rerunning continues where it left off,
because dedup and the idempotent session id mean the already-filed candidates
are skipped — that is what makes a ten-year history importable at all rather
than a single flood.
Dedup drops a candidate an approved claim or a pending proposal already
covers. Lexical first, deliberately: a base install has no
[embeddings]extra, and dedup that silently stops working is precisely how an unattended
import floods a review queue. The #147 embedding hits fold in on top when the
extra is present. Both halves are pinned — one test stubs the similarity module
so the fold-in runs, another forces the import to fail so the base-install path
is the one under test. CI installs
[dev,web], so without the second test thelexical-only path is the one nobody checks.
--no-dedupexists for a deliberate re-file;--dry-runreports withoutenqueuing;
--max-claims Nadditionally files receipt-backed claims from aconversation's answers.
the review gate
conversation_import.pyhas no path toapprove— asserted by a test thatreads the module's own imports, the same shape
test_capture_correction.pyuses. Everything lands PENDING and a human drains the queue as for any other
write.
acceptance criteria
vouch import <format> <path>emits pending proposals viapropose_*;no auto-approve
--dry-runreports counts without enqueuing--max-proposals Ncaps proposals per run and reports when the capwas hit
pass that works without the extra)
chat-jsonreuseschatgpt_import's parser andextract.extract_receipt_claims;markdown-vaultis the note-vault importer aliasedmake checkgreentests
50 in
tests/test_conversation_import.py: the propose-only invariant and theno-
approveimport check; each chat shape end to end plus the emptyconversation, the non-object entry, and the three unreadable-file refusals;
idempotent re-import, refresh-in-place on growth, and a decided proposal
blocking re-import; receipt verification on both the conversation-answer claims
and the memory claims; every memory-dump shape including the plain-text
fallback and within-dump repeats; dedup against an approved claim, against a
pending proposal, with the embedding path stubbed, with the extra absent, and
turned off; the cap on both importers, the rerun-continues property, and the
uncapped report; dry-run writing nothing; the byte ceilings for bare files and
zips; timestamp normalisation across seconds/millis/ISO/out-of-range; and the
CLI for every subcommand including a clean one-line error.
Full gate green locally:
pytest tests/ --ignore=tests/embeddings,mypy src,ruff check src tests, plusdiff-cover --fail-under 100on the changedlines.