FAQ | This is a LIVE service | Changelog

Skip to content
Snippets Groups Projects
  1. May 19, 2023
  2. Jan 24, 2023
    • J.W. Smith's avatar
      feat: Clearup functionality · bce461a1
      J.W. Smith authored
      Optional config argument to delete mid-process files at the end of a
      successful job. There is an associated command-line flag '--clearup'.
      
      The reason for this feature is to provide a way to slim down data
      volumes of production runs. A lot of processing files are generated
      and production runs may only need to retain configuration, logs and
      outputs. Also, the inputs might be archived separately, so they
      wouldn't need to be retained in prod runs.
      
      Currently, files are moved to a hard-coded 'recycling bin' directory,
      and there is a minimal set of files set for clearup: only the deposition
      and environment tar files.
      
      This change requires some standard structure to outputs of
      processor_in_job functions, which would be more robust if
      a class structure is implemented.
      
      TODO: Implement a test for file clearup.
      bce461a1
  3. Jan 12, 2023
  4. Jan 11, 2023
Loading