At the moment, pipelines are split into stages but handling of intermediate files is messy and it's hard to modify pipelines. It would be better if:
- stages take generic parameters for input/output files, not predetermined file paths
- file paths are defined by the Pipeline object, or whatever the stages are being used by
- no wildcards - part from date-stamped files in BCBio, we should know what every file will be called
- this might make output_files.yaml redundant
- checks for whether a stage should run uses presence of a reporting app stage is used as well as presence of files, not instead
We should also be able to mock a dataset, patch executor, run a pipeline and assert what all the bash commands were.
We should also make Stage objects as lightweight as possible, ideally removing their access to Dataset - see #395 for reasons why.
This might be a good opportunity to look at sciluigi
At the moment, pipelines are split into stages but handling of intermediate files is messy and it's hard to modify pipelines. It would be better if:
We should also be able to mock a dataset, patch
executor, run a pipeline and assert what all the bash commands were.We should also make Stage objects as lightweight as possible, ideally removing their access to Dataset - see #395 for reasons why.
This might be a good opportunity to look at sciluigi