making gitignore ignore the contents of the log dir rather than the dir itself. Think it may be why the integration tests are failing

This allows the ODK export functionality to append latest changes
to a single database path, instead of downloading the full database
history which can stretch over many years. This saves a lot of time
for the daily survey job.

The WRSIS survey processor can already limit the download size to
the current season(s) with a start-date parameter.

The Kobotoolbox survey processor doesn't have a solution for
avoiding the daily download of extensive historical data. This needs
investigating further at some point.

This will be helpful in cases when a given day's source calculation
fails as it allows the NAME spore dispersion model to run with the
previous day's calculation, provided there are no new surveys on
the latest day (which will only be known about once there is a
successful download again).

This reverts commit d84d1b1e.

some additional comments and code-tidying

See EpiModel commit 728921c.

moving the check for the "live" flag into the runProcess function, so will be picked up when not running directly from the command line

This is required for the standard epi model output, which has units of
ha_infected per ha_cell, because the model risk thresholds are fitted
for units of ha_infected per ha_wheat.

Optional config argument to delete mid-process files at the end of a
successful job. There is an associated command-line flag '--clearup'.

The reason for this feature is to provide a way to slim down data
volumes of production runs. A lot of processing files are generated
and production runs may only need to retain configuration, logs and
outputs. Also, the inputs might be archived separately, so they
wouldn't need to be retained in prod runs.

Currently, files are moved to a hard-coded 'recycling bin' directory,
and there is a minimal set of files set for clearup: only the deposition
and environment tar files.

This change requires some standard structure to outputs of
processor_in_job functions, which would be more robust if
a class structure is implemented.

TODO: Implement a test for file clearup.

chore: Tidy imports

And reports are sorted by date.
And assumed disease prevalence is set to medium, not low.

Can be called with command line flag '-p Scraper'.
To use, the config file requires the following minimal entry (with
URL for South Asia):
    "Scraper" : {
	    "ProcessPreJob" : "do_nothing_return_true",
        "ProcessInJob" : "process_in_job_media_scraper",
        "ProcessEWSPlotting" : "do_nothing",
        "seasonStartString" : "20220101",
        "URL" : "https://arrcc-viewer.onrender.com/assets/sample_data/data.zip"
    },

changing the scp logic and the ssh file checking logic to use a different command if no server name is set

This is necessary for Win to be able to rename the file afterward.

Before this commit, the process_pre_job_server_download() function would
check if the necessary tar file exists on the remote server. But, because it is
 written there by sftp, and sftp doesn't generate any lock file or temp files, in-progress sftp writes look like they are available and the job continues. With this change, the job will only process if the tar file is completely written.

restoring the original sys.exit to ProcessorUtils, dealing with exit commands in the unit tests by catching them

Because of changes to EpiPrepLister in the last few months, the file listers have changed name and are too complicated to guess. Now, it will be necessary for the user to specify them in the input config file. EpiPrepLister's docstring will have to guide the user clearly enough.