Optional config argument to delete mid-process files at the end of a
successful job. There is an associated command-line flag '--clearup'.

The reason for this feature is to provide a way to slim down data
volumes of production runs. A lot of processing files are generated
and production runs may only need to retain configuration, logs and
outputs. Also, the inputs might be archived separately, so they
wouldn't need to be retained in prod runs.

Currently, files are moved to a hard-coded 'recycling bin' directory,
and there is a minimal set of files set for clearup: only the deposition
and environment tar files.

This change requires some standard structure to outputs of
processor_in_job functions, which would be more robust if
a class structure is implemented.

TODO: Implement a test for file clearup.

Can be called with command line flag '-p Scraper'.
To use, the config file requires the following minimal entry (with
URL for South Asia):
    "Scraper" : {
	    "ProcessPreJob" : "do_nothing_return_true",
        "ProcessInJob" : "process_in_job_media_scraper",
        "ProcessEWSPlotting" : "do_nothing",
        "seasonStartString" : "20220101",
        "URL" : "https://arrcc-viewer.onrender.com/assets/sample_data/data.zip"
    },

changing the scp logic and the ssh file checking logic to use a different command if no server name is set

restoring the original sys.exit to ProcessorUtils, dealing with exit commands in the unit tests by catching them

The pre-job is meant to exit quietly if the prerequisite data is not
available yet. However, the epi pre-job check was raising an error
if AlternativeSuccessFileTemplate was missing from the config for
either Deposition or Environment. This is now handled quietly too.
CAUTION: This means the epi component will never raise an error
for missing input data, i.e., no email will ever say epi has failed to
run.

1) There is now a python alternative to the R-based wheat-source-
generation, called source_gen. The functionality to call source_gen
has been added (see ProcessorSurveys.pyL939 and L1032-1040), but not
yet switched on while we await testing of
output at the Met Office.

2) ProcessorComponents was huge (2100 lines!) and the structure was
not apparent. Each component's supporting functions are now in a
separate directory. There is also a ProcessorServer.py relating to
file down/up-loads. ProcessorComponents is now essentially a directory
 to import all the functions. This might inspire a more systematic
 (object-oriented?) structure for each component within coordinator
 in the future.

reading NAME.nc files rather than NAME.txt files (#19)

A useful function in ProcessorUtils is subprocess_and_log, which builds a subprocess command and write its stdout and stderr to the log handler. Beforehand, failures only ever generated an error message, but there are some use cases where another log type is preferred. This adds a log type option to the function, and uses it in one case where a warning is more appropriate.

Initial commit. Currently only working for Nepal deposition and (partially for) Ethiopia environment v2.0