fetch_wasabi_file doesn't check checksum after downloading new files

[ckarwin]

I noticed some strange behavior with fetch_wasabi_file when the user specifies the checksum. If the file is not present, it will download without a problem even if the downloaded file has the wrong checksum. This is because the checksum is never checked. But if you run when the file is already present, it then fails because it compares the checksum. An easy patch for this would be to compare the checksum after downloading the file for the first time, if a checksum is provided by the user.

The relevant part of the code is here:

cosipy/cosipy/util/data_fetching.py

Line 245 in 6d8a36e

s3.download_file(Bucket=bucket, Key=file, Filename=output)

[augustus-thomas]

Hey @ckarwin , I would like to help with this issue! I am just wanting to clarify my understanding:

Current Behavior

• If the file is not present and the provided checksum is incorrect, data_fetching succeeds and the file is downloaded.
• If the file is present and the provided checksum is incorrect, data_fetching fails.

Preferred Behavior

• If the file is not present and the provided checksum is incorrect, data_fetching should behave differently. My question is would you like it raise an exception and refuse to download the file, or download the file while raising a warning that the checksums do not match?
• If the file is present and the provided checksum is incorrect, data_fetching should still fail.

Would that be accurate?

[ckarwin]

Thanks, @augustus-thomas.

Your understanding of the current behavior is correct.

If the file is not present and the provided checksum is incorrect, data_fetching should behave differently. My question is would you like it raise an exception and refuse to download the file, or download the file while raising a warning that the checksums do not match?

You will need to download the file by default b/c it's the only practical way to get the checksum. If the user provides an incorrect checksum, an exception should be raised.

If the file is present and the provided checksum is incorrect, data_fetching should still fail.

Yes.

[israelmcmc]

You will need to download the file by default b/c it's the only practical way to get the checksum.

I agree. Just to add to this, Wasabi, in general, doesn’t store the md5sum. There’s a convoluted checksum I used to compare an existing file with one in Wasabi (see the get_size_and_etag method), which sometimes equals the md5sum but it’s not guaranteed. Also, if unzip=True, the checksum corresponds to the unzipped file, not the one in wasabi. And, even if there was a way, it’s good practice to compare checksum after downloading a file to make sure there were no issues during the transfer.