moving the check for the "live" flag into the runProcess function, so will be picked up when not running directly from the command line

This is required for the standard epi model output, which has units of
ha_infected per ha_cell, because the model risk thresholds are fitted
for units of ha_infected per ha_wheat.

Optional config argument to delete mid-process files at the end of a
successful job. There is an associated command-line flag '--clearup'.

The reason for this feature is to provide a way to slim down data
volumes of production runs. A lot of processing files are generated
and production runs may only need to retain configuration, logs and
outputs. Also, the inputs might be archived separately, so they
wouldn't need to be retained in prod runs.

Currently, files are moved to a hard-coded 'recycling bin' directory,
and there is a minimal set of files set for clearup: only the deposition
and environment tar files.

This change requires some standard structure to outputs of
processor_in_job functions, which would be more robust if
a class structure is implemented.

TODO: Implement a test for file clearup.

chore: Tidy imports

And reports are sorted by date.
And assumed disease prevalence is set to medium, not low.

Can be called with command line flag '-p Scraper'.
To use, the config file requires the following minimal entry (with
URL for South Asia):
    "Scraper" : {
	    "ProcessPreJob" : "do_nothing_return_true",
        "ProcessInJob" : "process_in_job_media_scraper",
        "ProcessEWSPlotting" : "do_nothing",
        "seasonStartString" : "20220101",
        "URL" : "https://arrcc-viewer.onrender.com/assets/sample_data/data.zip"
    },

changing the scp logic and the ssh file checking logic to use a different command if no server name is set

This is necessary for Win to be able to rename the file afterward.

Before this commit, the process_pre_job_server_download() function would
check if the necessary tar file exists on the remote server. But, because it is
 written there by sftp, and sftp doesn't generate any lock file or temp files, in-progress sftp writes look like they are available and the job continues. With this change, the job will only process if the tar file is completely written.

restoring the original sys.exit to ProcessorUtils, dealing with exit commands in the unit tests by catching them

Because of changes to EpiPrepLister in the last few months, the file listers have changed name and are too complicated to guess. Now, it will be necessary for the user to specify them in the input config file. EpiPrepLister's docstring will have to guide the user clearly enough.

This relates to developments to ease the maintenance
of the forecast system. With this development, there is no R code to
actively maintain. Moving from git project wheat-source-generation to
ews-source-generation. The python code has been validated for
identical performance to the R version. The only differences are
small changes from the underlying shapefile reading packages
(see notes-on-rewriting-in-python.txt in ews-source-generation).

Keeping the option to used the R code for some time in case any
follow-up investigation is required.