It is possible to specify which spatial points file to use in the main config json. The default file will be used if not specified. This will allow us to perfrom historic runs with the old grid and specify manually.
The country-level points filtering used by the postprocessing steps (beofre plotting) now round down the acuracy when filtering points. This allows the country filtering to work with both the new and the old grid, as the differece is lost in the rounding

So that coordinator tests work with ews-epidemiology commit d344567 .

An error once happened where it seems the request content was lost
before it could be used. This may mitigate the risk of it happening
again.

This is an initial step to have multiple source file setup.
This step is NOT utilising the grouped files yet.

Survey processor to handle WheatRustToolbox download and formating.

New function to allow conversion of a series according to a provided dictionary. (Needed for WRT formating.)

- Always create ExportCSV folder if it is not exist to allow merging.
- Correcting messages to not have ODK as the default download.
- Add ProcessorSurveysWRT import.

moving log file output to before the assert statement, so we can more easily see which file is missing [if a file is missing]

making gitignore ignore the contents of the log dir rather than the dir itself. Think it may be why the integration tests are failing

This allows the ODK export functionality to append latest changes
to a single database path, instead of downloading the full database
history which can stretch over many years. This saves a lot of time
for the daily survey job.

The WRSIS survey processor can already limit the download size to
the current season(s) with a start-date parameter.

The Kobotoolbox survey processor doesn't have a solution for
avoiding the daily download of extensive historical data. This needs
investigating further at some point.

This will be helpful in cases when a given day's source calculation
fails as it allows the NAME spore dispersion model to run with the
previous day's calculation, provided there are no new surveys on
the latest day (which will only be known about once there is a
successful download again).

This reverts commit d84d1b1e.

some additional comments and code-tidying

See EpiModel commit 728921c.

moving the check for the "live" flag into the runProcess function, so will be picked up when not running directly from the command line

This is required for the standard epi model output, which has units of
ha_infected per ha_cell, because the model risk thresholds are fitted
for units of ha_infected per ha_wheat.

Optional config argument to delete mid-process files at the end of a
successful job. There is an associated command-line flag '--clearup'.

The reason for this feature is to provide a way to slim down data
volumes of production runs. A lot of processing files are generated
and production runs may only need to retain configuration, logs and
outputs. Also, the inputs might be archived separately, so they
wouldn't need to be retained in prod runs.

Currently, files are moved to a hard-coded 'recycling bin' directory,
and there is a minimal set of files set for clearup: only the deposition
and environment tar files.

This change requires some standard structure to outputs of
processor_in_job functions, which would be more robust if
a class structure is implemented.

TODO: Implement a test for file clearup.

chore: Tidy imports

And reports are sorted by date.
And assumed disease prevalence is set to medium, not low.

Can be called with command line flag '-p Scraper'.
To use, the config file requires the following minimal entry (with
URL for South Asia):
    "Scraper" : {
	    "ProcessPreJob" : "do_nothing_return_true",
        "ProcessInJob" : "process_in_job_media_scraper",
        "ProcessEWSPlotting" : "do_nothing",
        "seasonStartString" : "20220101",
        "URL" : "https://arrcc-viewer.onrender.com/assets/sample_data/data.zip"
    },