diff --git a/docs/datafed/globus.md b/docs/datafed/globus.md index 3f5df06..5e2be0e 100644 --- a/docs/datafed/globus.md +++ b/docs/datafed/globus.md @@ -221,6 +221,29 @@ After JISC made some of our test datasets available via Globus, we ran several t Transfer rates were in the same range as those already observed between the other sites. Notably, and slightly surprisingly, the largest dataset (50x100GB files) was generally slightly slower to transfer than the two smaller datasets, although in most cases the speed difference was relatively small. +## Partial Data Transfers + +To test the case where a dataset is updated after its initial transfer, or where additional data is added to it later on, I performed some experiments with Globus to determine how it performs in this case. The experiments were as follows: + +1. Start with a dataset consisting of 10x100GB files (these were the first ten files from our standard 50x100GB dataset) on COSMA. +2. Transfer the entire dataset to another site (ARCHER2) using Globus. +3. Remove two of the files from ARCHER2 and transfer the dataset again. +4. Replace one file on COSMA with different contents and transfer the dataset again. +5. Add two additional files on COSMA and transfer the dataset again. + +By default, Globus will transfer all of the selected files from source to destination, overwriting any files already present at the destination, even if their content is the same. To handle partial transfers more intelligently it is necessary to check the "apply sync level" box under "Transfer & Timer Options" in the Globus web app. Several different sync levels are available, based on file size, modification time and checksum. For these tests I chose level 2, which will transfer any files with a different file size or a newer modification time on the source machine than on the destination. + +The data transfer times observed are recorded in the table below: + +| Test | Transfer time (minutes and seconds) | +| --- | --- | +| Initial transfer of entire dataset | 22:54 | +| Transfer with two files removed from destination | 9:18 | +| Transfer with one source file's contents replaced | 11:35 | +| Transfer with two additional files at source | 9:00 | + +Modifying the sync level setting clearly saves time as none of the partial transfers took as long as the initial transfer of the entire dataset. However, they did take proportionally longer for the amount of data being transferred. For example, the second transfer only had two files to copy but took around 40% of the time taken by the initial full transfer of ten files rather than the 20% that might have been expected based on data volumes. This could be because transferring only one or two files affords fewer opportunities for parallelism than transferring ten files at once. + ## Globus Command Line Interface Usage Globus transfers can be initiated and monitored through the Globus web interface available at [https://app.globus.org](https://app.globus.org). However, for some use cases (particularly when scripting and automation are desired), the command line interface may be more suitable. This can easily be installed through `pip`: diff --git a/docs/datafed/rsync.md b/docs/datafed/rsync.md index 2821d02..25f9355 100644 --- a/docs/datafed/rsync.md +++ b/docs/datafed/rsync.md @@ -9,10 +9,10 @@ Rsync is a standard tool for data transfer, and widely used for small volumes of | | Durham | Edinburgh | Cambridge | RAL | | --- | --- | --- | --- | --- | -| Durham | | 1150MB/s | 491MB/s | 71MB/s | -| Edinburgh | 1143MB/s | | 585MB/s | 94MB/s | -| Cambridge | 331MB/s | 566MB/s | | 52MB/s | -| RAL | 99MB/s | 128MB/s | 72.8MB/s | | +| Durham | | 1150MB/s | 491MB/s | 95MB/s | +| Edinburgh | 1143MB/s | | 585MB/s | 98MB/s | +| Cambridge | 331MB/s | 566MB/s | | 93MB/s | +| RAL | 111MB/s | 128MB/s | 100MB/s | | (from location in column to location in row). @@ -87,3 +87,43 @@ The transfers involving JASMIN were performed in a similar way, but used a much | CSD3 | JASMIN | 46.0 | These transfers were much slower than Globus, which was typically able to transfer this small dataset in or out of JASMIN at a rate of 200-800MB/s. This suggests that rsync may have a larger start up overhead than Globus and therefore perform more poorly on smaller transfers that do not run for long enough to effectively amortise this cost. To test this hypothesis, a transfer of the small 90GB dataset from COSMA to ARCHER2 was performed. This ran at a rate of 117.2MB/s, much slower than the roughly 1GB/s observed with the larger dataset between those systems. + +The initial JASMIN tests were run from a standard login node. Further tests were later run from a dedicated data transfer VM at the site. These are shown in the table below. They mostly show slightly, but not dramatically, faster transfer rates than those obtained using the login node. + +| Source | Destination | Transfer rate (MB/s) | +| ----------- | ----------- | ----------- | +| JASMIN | COSMA | 94.5 | +| COSMA | JASMIN | 109.6 | +| JASMIN | COSMA | 73.6 | +| COSMA | JASMIN | 111.0 | +| JASMIN | CSD3 | 75.0 | +| CSD3 | JASMIN | 99.7 | +| JASMIN | CSD3 | 92.8 | +| CSD3 | JASMIN | 68.8 | +| JASMIN | ARCHER2 | 97.7 | +| ARCHER2 | JASMIN | 111.4 | +| JASMIN | ARCHER2 | 97.0 | +| ARCHER2 | JASMIN | 119.5 | + +## Partial transfers + +rsync is known for its ability to synchronise data at its source and destination by intelligently transferring only the modified data rather than re-transferring the whole dataset. I tested this by performing some simple experiments: + +1. Start with a dataset consisting of 10x100GB files (these were the first ten files from our standard 50x100GB dataset) on COSMA. +2. Transfer the entire dataset to another site (ARCHER2) using rsync. +3. Remove two of the files from ARCHER2 and transfer the dataset again. +4. Replace one file on COSMA with different contents and transfer the dataset again. +5. Add two additional files on COSMA and transfer the dataset again. + +By default, rsync will use a file's modification timestamp to determine whether the file needs to be transferred or not. If this does not match between source and destination, the file will be transferred again. Note that even if the timestamp is newer at the destination, this will not be regarded as a match and the file will be transferred again. Therefore, to make sure the timestamps do match, it is necessary to pass the `-t` option to rsync when performing the initial transfer of the entire dataset. There is also an option available to checksum the files rather than relying on timestamps, but I did not enable this. + +The data transfer times observed are recorded in the table below: + +| Test | Transfer time (minutes and seconds) | +| --- | --- | +| Initial transfer of entire dataset | 135:32 | +| Transfer with two files removed from destination | 27:18 | +| Transfer with one source file's contents replaced | 231:29 | +| Transfer with two additional files at source | 14:22 | + +As expected, the second and fourth transfers were much faster than the initial transfer, indicating that rsync only transferred the modified files rather than all of the files. Interestingly, the third transfer, with one source file's contents being replaced, was much slower than even the initial full transfer, despite the fact that the output of the rsync command indicated that only the one modified file was transferred. This finding was reproduced multiple times on different days, suggesting that it was not due to a temporarily slow network link. I suspect that the slowness was due to rsync reading both source and destination files and comparing blocks to determine which parts of the file have actually changed, and only transferring the portions that have. However, in this case the entire file had changed, so this procedure would have added significant overhead for no gain. The degree to which this slowed the transfer down is surprising.