feat: Clearup functionality
Optional config argument to delete mid-process files at the end of a successful job. There is an associated command-line flag '--clearup'. The reason for this feature is to provide a way to slim down data volumes of production runs. A lot of processing files are generated and production runs may only need to retain configuration, logs and outputs. Also, the inputs might be archived separately, so they wouldn't need to be retained in prod runs. Currently, files are moved to a hard-coded 'recycling bin' directory, and there is a minimal set of files set for clearup: only the deposition and environment tar files. This change requires some standard structure to outputs of processor_in_job functions, which would be more robust if a class structure is implemented. TODO: Implement a test for file clearup.
Showing
- coordinator/Processor.py 57 additions, 17 deletionscoordinator/Processor.py
- coordinator/ProcessorAdvisory.py 7 additions, 1 deletioncoordinator/ProcessorAdvisory.py
- coordinator/ProcessorDeposition.py 7 additions, 1 deletioncoordinator/ProcessorDeposition.py
- coordinator/ProcessorEnvironment.py 14 additions, 2 deletionscoordinator/ProcessorEnvironment.py
- coordinator/ProcessorEpidemiology.py 8 additions, 2 deletionscoordinator/ProcessorEpidemiology.py
- coordinator/ProcessorScraper.py 7 additions, 1 deletioncoordinator/ProcessorScraper.py
- coordinator/ProcessorSurveys.py 7 additions, 1 deletioncoordinator/ProcessorSurveys.py
- coordinator/ProcessorUtils.py 45 additions, 1 deletioncoordinator/ProcessorUtils.py
Loading
Please register or sign in to comment