This is moves field area calculations out of the source_gen package and into the server-specific load function.

Not for immediate production, awaiting verification from Yoseph.

Credentials to access the new survey form are available on episerver at
/storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/
Cred-ODK-EIAR_newODK.json

Adding the functionality to group surveys and run source gen on each groups. Grouping is only run if it is defined in the config (otherwise it use all available surveys), and it is based on the 'Origin' column of the surveys. All unique 'Origin' will create its own group alongside the ones defined in the config. Groups can be ignored by defining 'GroupsToIgnore' in the config (this has no effect on 'Origin' types).

Handle missing redundant columns and continue to set a common column
order at the same time.

This is useful for the new ODK form until the wheat rust toolbox is setup.

This is an initial step to have multiple source file setup.
This step is NOT utilising the grouped files yet.

- Always create ExportCSV folder if it is not exist to allow merging.
- Correcting messages to not have ODK as the default download.
- Add ProcessorSurveysWRT import.

This will be helpful in cases when a given day's source calculation
fails as it allows the NAME spore dispersion model to run with the
previous day's calculation, provided there are no new surveys on
the latest day (which will only be known about once there is a
successful download again).

This reverts commit d84d1b1e.

Optional config argument to delete mid-process files at the end of a
successful job. There is an associated command-line flag '--clearup'.

The reason for this feature is to provide a way to slim down data
volumes of production runs. A lot of processing files are generated
and production runs may only need to retain configuration, logs and
outputs. Also, the inputs might be archived separately, so they
wouldn't need to be retained in prod runs.

Currently, files are moved to a hard-coded 'recycling bin' directory,
and there is a minimal set of files set for clearup: only the deposition
and environment tar files.

This change requires some standard structure to outputs of
processor_in_job functions, which would be more robust if
a class structure is implemented.

TODO: Implement a test for file clearup.

This relates to developments to ease the maintenance
of the forecast system. With this development, there is no R code to
actively maintain. Moving from git project wheat-source-generation to
ews-source-generation. The python code has been validated for
identical performance to the R version. The only differences are
small changes from the underlying shapefile reading packages
(see notes-on-rewriting-in-python.txt in ews-source-generation).

Keeping the option to used the R code for some time in case any
follow-up investigation is required.

WRSIS HTTP response should have 'response' in it which contains the survey information, even if it is empty:
{'msg': 'successful', 'response': {'Rust Survey Data': []}, 'status': '200'}

The WRSIS download KEY column is an intger which made a problem in the survey data remove subprocess. The solution is to read the KEY column always as a string.

Date formating is necessery for the WRSIS API.

1) There is now a python alternative to the R-based wheat-source-
generation, called source_gen. The functionality to call source_gen
has been added (see ProcessorSurveys.pyL939 and L1032-1040), but not
yet switched on while we await testing of
output at the Met Office.

2) ProcessorComponents was huge (2100 lines!) and the structure was
not apparent. Each component's supporting functions are now in a
separate directory. There is also a ProcessorServer.py relating to
file down/up-loads. ProcessorComponents is now essentially a directory
 to import all the functions. This might inspire a more systematic
 (object-oriented?) structure for each component within coordinator
 in the future.