FAQ | This is a LIVE service | Changelog

Skip to content
Snippets Groups Projects
  • J.W. Smith's avatar
    bce461a1
    feat: Clearup functionality · bce461a1
    J.W. Smith authored
    Optional config argument to delete mid-process files at the end of a
    successful job. There is an associated command-line flag '--clearup'.
    
    The reason for this feature is to provide a way to slim down data
    volumes of production runs. A lot of processing files are generated
    and production runs may only need to retain configuration, logs and
    outputs. Also, the inputs might be archived separately, so they
    wouldn't need to be retained in prod runs.
    
    Currently, files are moved to a hard-coded 'recycling bin' directory,
    and there is a minimal set of files set for clearup: only the deposition
    and environment tar files.
    
    This change requires some standard structure to outputs of
    processor_in_job functions, which would be more robust if
    a class structure is implemented.
    
    TODO: Implement a test for file clearup.
    bce461a1
    History
    feat: Clearup functionality
    J.W. Smith authored
    Optional config argument to delete mid-process files at the end of a
    successful job. There is an associated command-line flag '--clearup'.
    
    The reason for this feature is to provide a way to slim down data
    volumes of production runs. A lot of processing files are generated
    and production runs may only need to retain configuration, logs and
    outputs. Also, the inputs might be archived separately, so they
    wouldn't need to be retained in prod runs.
    
    Currently, files are moved to a hard-coded 'recycling bin' directory,
    and there is a minimal set of files set for clearup: only the deposition
    and environment tar files.
    
    This change requires some standard structure to outputs of
    processor_in_job functions, which would be more robust if
    a class structure is implemented.
    
    TODO: Implement a test for file clearup.
Code owners
Assign users and groups as approvers for specific file changes. Learn more.