Data integrity

[arita37]

Hello,

Some Question

1. how the data is stored in the remote S3 storage ?
   Compressed of original files or custom format

2. For some reasons zerofs failed copy (ie process shutdown),
   how data integrity is preserved ?
   how to recover existing files directly ?

Think the main challenge of using this tool is the data integrity/recovery after failure.

Imagine you have 10 TB of data...
and they are not accessible after write failure.... (!)

[Barre]

Hi,

1. Data is stored in a LSM tree.
2. There should not be any corruption after unclean shutdown, otherwise that's a bug.

[arita37]

Yes, we need actual reliability tessts
when client crashing
as well recovery scenarios.

Otherwise, we will never use this framework
outside of toy demo or data which be lost….

reliability during crash is more important
than encryption….

Encryption: most of the data are not encrypted
or already encrypted at the S3 level…
so a bit useless….

Believe me, am in enterprise setting

[arita37]

For me, if no reliability and crash test,
this project is still at infancy/early stage.

[Barre]

You are more than welcome to contribute!