Fjall is an excellent key-value database, but I didn't fully evaluate it during the initial selection. Since switching from 0.3.7 to the 1.x series as our blob storage backend, we've received numerous issue reports—both during upgrades and in day-to-day maintenance. Many of these issues are difficult to resolve, and it feels like we've fallen into a trap where fixing one problem introduces several more.
Fjall doesn't seem well-suited for email storage, especially with large messages and large attachments. To better address compatibility issues and provide a more stable, durable storage layer, I've had to implement the storage backend myself. This wasn't something I wanted to do—it's not my area of expertise—but I haven't found an ideal off-the-shelf solution.
Storing each email as a single .eml file on the filesystem would mean managing millions or even hundreds of millions of files. At TB-scale, that's a huge strain on the filesystem. Systems based on Bitcask or WiscKey either lack mature Rust libraries or require external storage services, and their implementations may not fit email storage well either. I want Bichon to manage its own storage without relying on external databases.
The bichon-blob implementation (see README in the crate) essentially appends each email to a segment file with per-message compression, and stores position metadata in Redb. This minimizes index memory consumption, with the goal of supporting TB-scale storage on a single instance.
To avoid the widespread upgrade issues we saw with the 0.3.7 → 1.x migration, we're doing a small-scale preview this time. We'll collect feedback, fix issues, and then officially release the 2.x version.
What this branch includes:
bichon-admin now offers both 0.3.7 → 2.x and 1.x → 2.x upgrade paths- The
0.3.7 → 1.x upgrade option is now deprecated (but existing 1.x data remains intact)
- Old data is never deleted — this means the upgrade requires double the disk space for your existing data
Important notes:
- No pre-built binaries are provided for this preview — you'll need to clone the branch and compile it yourself
- The
1.x → 2.x upgrade does not rebuild indexes, so memory consumption is significantly lower than the 0.3.7 → 1.x upgrade
- I've tested both upgrade paths internally and they've been smooth — no issues encountered so far
I'd love to get feedback from users with large datasets, especially those who ran into trouble during the previous 0.3.7 → 1.x upgrade. Your input would be invaluable.
Perhaps this implementation still isn't perfect, but I believe I can more easily fix any issues that arise — and it's far more controllable, since the storage code is now part of Bichon itself.
I truly appreciate the community's support and patience throughout this journey. My goal is to deliver a long-term, stable solution so that we don't have to put users through manual migration and upgrades again and again.
If you run into any issues during testing, please reply directly in this issue thread. I'll be monitoring closely and will help troubleshoot as quickly as possible.
Thank you all so much for your support — it truly means a lot to me and to the project. 🙏
Fjall is an excellent key-value database, but I didn't fully evaluate it during the initial selection. Since switching from 0.3.7 to the 1.x series as our blob storage backend, we've received numerous issue reports—both during upgrades and in day-to-day maintenance. Many of these issues are difficult to resolve, and it feels like we've fallen into a trap where fixing one problem introduces several more.
Fjall doesn't seem well-suited for email storage, especially with large messages and large attachments. To better address compatibility issues and provide a more stable, durable storage layer, I've had to implement the storage backend myself. This wasn't something I wanted to do—it's not my area of expertise—but I haven't found an ideal off-the-shelf solution.
Storing each email as a single
.emlfile on the filesystem would mean managing millions or even hundreds of millions of files. At TB-scale, that's a huge strain on the filesystem. Systems based on Bitcask or WiscKey either lack mature Rust libraries or require external storage services, and their implementations may not fit email storage well either. I want Bichon to manage its own storage without relying on external databases.The
bichon-blobimplementation (see README in the crate) essentially appends each email to a segment file with per-message compression, and stores position metadata in Redb. This minimizes index memory consumption, with the goal of supporting TB-scale storage on a single instance.To avoid the widespread upgrade issues we saw with the 0.3.7 → 1.x migration, we're doing a small-scale preview this time. We'll collect feedback, fix issues, and then officially release the 2.x version.
What this branch includes:
bichon-adminnow offers both0.3.7 → 2.xand1.x → 2.xupgrade paths0.3.7 → 1.xupgrade option is now deprecated (but existing 1.x data remains intact)Important notes:
1.x → 2.xupgrade does not rebuild indexes, so memory consumption is significantly lower than the0.3.7 → 1.xupgradeI'd love to get feedback from users with large datasets, especially those who ran into trouble during the previous
0.3.7 → 1.xupgrade. Your input would be invaluable.Perhaps this implementation still isn't perfect, but I believe I can more easily fix any issues that arise — and it's far more controllable, since the storage code is now part of Bichon itself.
I truly appreciate the community's support and patience throughout this journey. My goal is to deliver a long-term, stable solution so that we don't have to put users through manual migration and upgrades again and again.
If you run into any issues during testing, please reply directly in this issue thread. I'll be monitoring closely and will help troubleshoot as quickly as possible.
Thank you all so much for your support — it truly means a lot to me and to the project. 🙏