Something went wrong on our end
-
J.W. Smith authored
Optional config argument to delete mid-process files at the end of a successful job. There is an associated command-line flag '--clearup'. The reason for this feature is to provide a way to slim down data volumes of production runs. A lot of processing files are generated and production runs may only need to retain configuration, logs and outputs. Also, the inputs might be archived separately, so they wouldn't need to be retained in prod runs. Currently, files are moved to a hard-coded 'recycling bin' directory, and there is a minimal set of files set for clearup: only the deposition and environment tar files. This change requires some standard structure to outputs of processor_in_job functions, which would be more robust if a class structure is implemented. TODO: Implement a test for file clearup.
J.W. Smith authoredOptional config argument to delete mid-process files at the end of a successful job. There is an associated command-line flag '--clearup'. The reason for this feature is to provide a way to slim down data volumes of production runs. A lot of processing files are generated and production runs may only need to retain configuration, logs and outputs. Also, the inputs might be archived separately, so they wouldn't need to be retained in prod runs. Currently, files are moved to a hard-coded 'recycling bin' directory, and there is a minimal set of files set for clearup: only the deposition and environment tar files. This change requires some standard structure to outputs of processor_in_job functions, which would be more robust if a class structure is implemented. TODO: Implement a test for file clearup.
Code owners
Assign users and groups as approvers for specific file changes. Learn more.