File integrity check

[crazyscot]

While QUIC provides integrity protection for all data transferred, some sort of hash or checksum may provide extra confidence and value:

• in ensuring that qcp didn't mess anything up when we start moving multiple files around (Support for copying multiple files #21)
• if we were to investigate split-streaming file transfers
• as a "checkpoint" on exceptionally large file transfers
• in some industries where checksums are already a required part of the workflow (e.g. film rushes - ASC MHL reports)

A correspondent writes:

[...] one of the thoughts in people's minds might be: "great, it's secure, but how do I know about integrity/corruption? I thought TCP took care of that and UDP is lossy, how do I know that pieces aren't missing? Or that with UDP not guaranteeing ordering, maybe parts of my files are swapped around?"

The obvious way to do this with the current implementation is to pass file data through a digest algorithm on both sides as we go, and compare notes afterwards. However:

• if we were to split-stream files, this wouldn't work on its own and a more nuanced checksum-per-chunk would be required.
  On particularly large file transfers, there could be value in a sort of "checkpointing"

Checking could be optional; we might offer a choice of algorithms. For example:

• sha256 (max file size 2^64 bits or 2.3EB)
• xxhash - not cryptographically secure but stronger than crc64 and much faster than the sha2 family
• crc64

Or we might have it always-on, defaulting to a cheap algorithm with the option to add cryptographic strength if desired.

How we handle any checksum mismatch is very important, this will need to be thought through properly. Behaviour might be configurable.

We might even offer to output a file transfer/checksum report. This could be in a variety of formats, including ASC MHL.

Work to be done

• Establish initial algorithm library (would like pure-rust implementations, preferably fast so as not to impact transfer speed)
• Understand the semantics. Obvious the sender can checksum as it transfers, but where does the receiver checksum? This might be configurable (on receipt from the wire vs read-back from storage).
• Design protocol changes for both directions of transfer
• Build into CLI
• Figure out how we test checksum failures end-to-end. Perhaps a hidden (default off!) cfg option would enable a configuration or or environment variable that induced failures with a given probability.
• Investigate output report formats (this might spin into a separate ticket)

Protocol ideas

• FileHeader adds an enum for the checksum algorithm in use and any additional options
• FileTrailer adds the digest: enum (to identify) and byte array

[terrorbyte]

You might also want to look at BLAKE2b or BLAKE2s, which seems to be pretty popular for accelerated algorithms with cryptographically secure properties (more of an integrity check but still). It might also be possible to do be creative and do stream of hashes based on a block size vs the whole block, ie sign blocks of 10MB at a time and verify on both ends and allow the block size to be configurable to prevent having to do it all at once and potentially catch the errors early and even recover.

BLAKE2b is also used in librsync and 2s is a primitive in WireGuard for it's speed properties.

(every day I wish we lived in a world where we could rely on underlying fs hashes like ZFS/Lustre world, it would be so much easier to just use the underlying file integrity checks, but alas haha)