`-t list.txt` assigns wrong local file name to remote s3 content

[WillForan]

We get different files depending on the usage of the -t list.txt switch.

python version 2.7.5 on centOS 7.5.1804

./test_dl.bash
## With -t
('downloaded: ', 's3://NDAR_Central_2/submission_17025/t2/P0125_t2.zip')
('downloaded: ', 's3://NDAR_Central_2/submission_17025/t1/P0124_t1.zip')
 ('downloaded: ', 's3://NDAR_Central_1/submission_17020/localizers/P0125_localizers.zip')
('downloaded: ', 's3://NDAR_Central_2/submission_17025/diffusion/P0124_mr_diffusion.zip')

## Individual
('downloaded: ', 's3://NDAR_Central_2/submission_17025/t2/P0125_t2.zip')
('downloaded: ', 's3://NDAR_Central_1/submission_17020/localizers/P0125_localizers.zip')
('downloaded: ', 's3://NDAR_Central_2/submission_17025/t1/P0124_t1.zip')
('downloaded: ', 's3://NDAR_Central_2/submission_17025/diffusion/P0124_mr_diffusion.zip')

Binary files all_at_once/submission_17025/t2/P0125_t2.zip and one_at_a_time/submission_17025/t2/P0125_t2.zip differ
Binary files all_at_once/submission_17025/t1/P0124_t1.zip and one_at_a_time/submission_17025/t1/P0124_t1.zip differ
Binary files all_at_once/submission_17025/diffusion/P0124_mr_diffusion.zip and one_at_a_time/submission_17025/diffusion/P0124_mr_diffusion.zip differ
P0125_localizers.zip same in both

---

we were clued into this because the archive extracted names do not match the archive in the -t version. Here we were expecting t2 but got localizers when using the txt list option

$ for z in {all_at_once,one_at_a_time}/submission_17025/t2/P0125_t2.zip; do unzip -l $z|sed 5q; done                                                                                           
Archive:  all_at_once/submission_17025/t2/P0125_t2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  10-02-2018 18:52   P0125/P0125_localizers/
        0  10-02-2018 18:52   P0125/P0125_localizers/data/

Archive:  one_at_a_time/submission_17025/t2/P0125_t2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  10-15-2018 20:30   P0125/P0125_t2/
        0  10-15-2018 20:30   P0125/P0125_t2/data/

---

test_dl.bash

# setup
read u p <<<$(cat txt/cred.txt)
export PYTHONPATH="$PYTHONPATH:nda_aws_token_generator/python/" 
dl(){ python AWS-Download/awsdownload.py  -u "$u"  -p "$p" $@;}
mkdir -p all_at_once one_at_a_time

cat > list.txt <<HEREDOC
s3://NDAR_Central_2/submission_17025/t2/P0125_t2.zip
s3://NDAR_Central_1/submission_17020/localizers/P0125_localizers.zip
s3://NDAR_Central_2/submission_17025/t1/P0124_t1.zip
s3://NDAR_Central_2/submission_17025/diffusion/P0124_mr_diffusion.zip
HEREDOC

# two ways of getting files
echo "## With -t"
dl -t list.txt -d all_at_once
echo
echo "## Individual"
while read l; do dl -d one_at_a_time $l; done < list.txt

# compare
find one_at_a_time/ -iname '*zip'| while read z; do
  diff ${z/one_at_a_time/all_at_once} $z && echo "$(basename $z) same in both"
done

[WillForan]

It looks like maybe for each parallel pool, the same file is downloaded?
From ~800 subject s3 links we only extract ~260 unique subjects when the archives are downloaded using -t. (zip files have since been removed, cannot check for sure)

In the example above, all downloaded files are actually the localizer archive despite their name. Weirdly, localizer is the second in the list (but first if sorted). Removing all the files and rerunning does not change which file is actually downloaded.

for z in all_at_once/submission_170*/*/*.zip; do unzip -l $z|sed 4q; done| awk '(/P/){print $NF}'|paste - -|column -t

all_at_once/submission_17020/localizers/P0125_localizers.zip   P0125/P0125_localizers/
all_at_once/submission_17025/diffusion/P0124_mr_diffusion.zip  P0125/P0125_localizers/
all_at_once/submission_17025/t1/P0124_t1.zip                   P0125/P0125_localizers/
all_at_once/submission_17025/t2/P0125_t2.zip                   P0125/P0125_localizers/


for z in all_at_once/submission_170*/*/*.zip; do diff all_at_once/submission_17020/localizers/P0125_localizers.zip $z && echo $z is localizer; done

all_at_once/submission_17020/localizers/P0125_localizers.zip is localizer
all_at_once/submission_17025/diffusion/P0124_mr_diffusion.zip is localizer
all_at_once/submission_17025/t1/P0124_t1.zip is localizer
all_at_once/submission_17025/t2/P0125_t2.zip is localizer

[ushnaahmad]

Hi Will,

Thank you for reporting the issue. This was something we had been seeing previously as well, and you are right in that each worker thread was downloading the same file with a different name. This issue has since then been resolved. Can you confirm that you are running the most updated version of the script? The newest version was uploaded about a month ago, and with the new version, we were unable to replicate the issue you are seeing.

In P0125_t2.zip, I see /t2/P0125/P0125_t2/data/

Please try again with this version of the script and let me know if you continue to see the errors.

I would also like to direct you to nda-tools (https://pypi.org/project/nda-tools/), our python package for interacting with NDA web services. This package includes a preinstalled command line tool (downloadcmd) to download your NDA data, that has the same commands and functionality as this script. All updates and new features will be added to nda-tools.