BICAS: bicas.tools.batch should update datasets when parents are updated (enhancement/shortcoming)

[ErikPGJ]

Problem

bicas.tools.batch ("mode V01") generates datasets whenever it finds that
(1) a particular dataset could be generated (given a set of potential parents),
(2) the "child" dataset does not already exist.

If the child dataset already exists, but the parents are updated, then not new dataset is produced.

Potential solutions

• PROPOSAL: When determining which BPCIs to use for processing, then bicas.tools.batch should look at the child and parent file modification dates (if there are pre-existing children). If the parents are younger than any of the children, then it should be used.

  • CON/NOTE: Syncing remote parent datasets could mess up the file modification dates (setting them to a recent date), triggering re-processing of all datasets.

• PROPOSAL: When determining which BPCIs to use for processing, then bicas.tools.batch should look at the parent versions and compare them with the parent versions in the child metadata (GAs).

  • CON: Must read every existing child dataset GAs. == > Likely slow
  • NOTE/CON: Relies on dataset GAs both (1) existing and (2) being correctly formatted.
    • CON: Old GA: Should be reliable.

• PROPOSAL: When determining which BPCIs to use for processing, then bicas.tools.batch should look at the parent generation time (GA) and compare them with the child generation time (GAs).

  • CON: Must read every existing parent and child dataset GA. == > Likely slow

• PROPOSAL: Cache GAs read to speed up algorithms. A unique dataset filename (or path) should be a unique key to the same GAs.