Investigate the usage of batch FTP transfer methods

[nicolasCtd]

The Vortex codebase contains several FTP transfer methods whose names include "batch", for example:

• ftserv_batchget
• batchrawftget
• smartbatchftget

These methods are intended to transfer multiple files in a single call, but it is currently unclear whether they are still used.

[nicolasCtd]

The script below shows how to transfer a list of files from Hendrix, with smartbatchftget.

Note that FTSERV will be used if "ftserv" is defined in the Vortex configuration. Otherwise, the standard ftget method will be used.

from vortex.tools.systems import OSExtended
import vortex
import os


class TestSystem(OSExtended):
    def __init__(self):
        super().__init__()

logname = "coutandn"
port = None
hostname = "hendrix.meteo.fr"

# Add "ftserv" to the Vortex configuration => ftserv_batchget() will be used.
# If false, the standard method ftget() will be used.
USE_FTSERV = True

sources = ["file1", "file2", "file3"]   # these files exist on the remote host
destinations = ["f1", "f2", "f3"] 

system = TestSystem()

for dst in destinations:
    path = system.path.abspath(dst)
    if os.path.exists(path):
        os.remove(path)

if USE_FTSERV:
    vortex.config.set_config("ftserv", "hostname_patterns", ["(belenos|taranis|sxcoope)transfert\\d.\\1hpc.meteo.fr|sxcoope1"])
    
system.smartbatchftget(
    source=sources,
    destination=destinations,
    hostname=hostname,
    logname=logname,
    port=port,
    cpipeline=None,
)

for dst in destinations:
    assert system.path.exists(dst), f"Transfer failed: {dst}"