Skip to content

Add complete, restart-safe UIDONLY acquisition - #334

Closed
gabeosx wants to merge 4 commits into
rustmailer:mainfrom
gabeosx:uidonly-acquisition
Closed

Add complete, restart-safe UIDONLY acquisition#334
gabeosx wants to merge 4 commits into
rustmailer:mainfrom
gabeosx:uidonly-acquisition

Conversation

@gabeosx

@gabeosx gabeosx commented Jul 29, 2026

Copy link
Copy Markdown

Summary

Add a separate UIDONLY acquisition path for servers that advertise RFC 9586,
while preserving Bichon's existing acquisition path for other servers.

The new path:

  • enables and confirms UIDONLY after every authentication and reconnect;
  • fixes each acquisition snapshot at UIDNEXT - 1;
  • inventories UIDs in bounded ascending PARTIAL pages;
  • records pending, committed, vanished, and failed state by endpoint, account,
    canonical mailbox, UIDVALIDITY, and UID;
  • fetches each planned UID within message, literal, total-byte, runtime,
    cancellation, and disk limits;
  • stores the complete raw message through Bichon's existing blob, envelope,
    attachment, and Tantivy path;
  • reads back and verifies the canonical record before committing the UID;
  • retries unresolved UIDs rather than checkpointing over them;
  • keeps distinct logical UID records when raw message bodies are identical;
  • reuses the same ledger for scheduled incremental acquisition; and
  • removes only the exact account/mailbox-owned staging state during existing
    lifecycle deletion.

Why

This implements the Bichon part of #333.

The existing full-mailbox path plans work from mutable message sequence
numbers. An expunge between pages can shift later sequence numbers and leave a
message behind. Existing progress tracking can also advance to a higher UID
after a lower message was omitted, oversized, interrupted, or only queued for
storage. Once the saved cursor moves above that message, later runs may never
request it again.

That mechanism is relevant to the incomplete Yahoo archives reported in #286
and #271, and to the interrupted-acquisition case in #255.

Dependency stack

This draft depends on:

For reproducible review, Bichon currently pins the exact feature commit
905a005234f89d046c2f439cee27daf947917b41 from gabeosx/async-imap, which in
turn pins parser commit 05439a90033d67297892c3fe206b8c4285df3821.
Those temporary sources should be replaced with the accepted Rustmailer commits
or releases before this PR merges.

Behavior worth calling out

  • Servers without UIDONLY continue through the existing Bichon path.
  • Once UIDONLY is enabled, this path never issues a sequence-number command.
  • A server that returns more than the requested PARTIAL page is rejected.
  • Huge VANISHED ranges are retained and intersected compactly rather than
    expanded into millions of UIDs.
  • A changed UIDVALIDITY starts a new staging epoch and does not reuse the old
    UID checkpoint.
  • Date-filtered UIDONLY acquisition currently fails closed because a
    date-bounded UID inventory has not yet been defined. It will not silently
    claim a complete snapshot.

Tests

Against the exact dependency commits above:

  • cargo check -p bichon-core --no-default-features --offline --locked
  • UIDONLY module: 34 passed; 2 explicitly opt-in tests ignored
  • ordinary acquisition-flow regressions: 17 passed; 4 credentialed tests
    ignored
  • isolated production-storage exact-raw restore: 1 passed
  • disposable Cyrus 3.12.2 exact-byte acquisition and restart: 1 passed

The deterministic coverage includes expunge between inventory pages, sparse
UIDs, omitted and oversized messages, disconnect/reconnect, pending restart,
UIDVALIDITY changes, identical bodies, malformed or overfull PARTIAL responses,
MESSAGELIMIT handling, huge VANISHED ranges, pre-literal response ceilings,
runtime/cancellation/disk limits, canonical-write rollback, and lifecycle
cleanup.

The Cyrus harness builds a pinned, checksum-verified server, uses a random
localhost port and disposable volumes, provisions three synthetic messages,
and cleans up the container, volumes, image, and temporary archive when done.

A fresh GitHub clone of this branch also completed the locked build, fake-server
suite, exact-raw production-storage test, and a newly provisioned Cyrus run.

@gabeosx

gabeosx commented Aug 4, 2026

Copy link
Copy Markdown
Author

Superseded by #337. The replacement keeps the opt-in Yahoo UIDONLY implementation entirely within Bichon, uses the existing dependency versions, and continues tracking the proposal in #333.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants