The method in this example may become the official method of downoading Ethiopia wheat rust surveys. Based on proposed API provided by CSM on 23 September 2020.

A useful function in ProcessorUtils is subprocess_and_log, which builds a subprocess command and write its stdout and stderr to the log handler. Beforehand, failures only ever generated an error message, but there are some use cases where another log type is preferred. This adds a log type option to the function, and uses it in one case where a warning is more appropriate.

From 3rd February 2021 each Wednesday, Will at the Met Office will send deposition and env suit met calcs using analysis meteorology. The results can be compared with standard calcs using forecast meteorology. To get a simple reproduction of the layout, the analysis jobs are handled in a separate workspace. Crontab will be switched on after 27th Jan.

The tarred directory of files from the Met Office includes the NAME .txt files to use as input, as well as various log files and diagnostic plots. The number of log files changed recently but shouldn't affect the operatinal calcs. This change means the check for a complete transfer of incoming files cares only about the NAME .txt files.

Output was tested against EWS-Coordinator v1.1 and found to be the same. v1.1 and v2.0 were running alongside for two weeks to monitor the new system and it performed as well as v1.1.

This should only apply to the operational copy of the EWS-Coordinator. If you want to run a separate copy for any reason, make these changes to point to your separate workspace. The git ignored files still need pointing to, but they don't need to be duplicated. Your local version of these changes can be stashed while pushing and pulling to/from the master branch.

In operational version, this is needed for survey results. The way this is implemented, it can be configured for any other component too.

Updating Env Suit v2.0 calculation. Copied from operational EnvProcessor.py in master branch. Not tested for this commit.

Hard-coded check of length of exported survey data as a basic check of ODK Briefcase for robustness (in particular for exports from Ethiopia). Currently, 2715 is an expected minimum for Ethiopia. Any small increases in coming weeks are unlikely to lead to not catching an incomplete export. This check is not important for SouthAsia data export which has been reliable. Because of the large data size, this 2715 limit is almost ineffective for South Asia anyway.

Errors in process_pre_job can create a job directory to record the error (instead of rediscovering every time the script is called).

Update processing time for South Asia to match Ethiopia. Currently, Met Office scripts start at 9am.

Modified to run multiple config files for the same deposition region. Still an untidy method to handle parameters DiseaseName, RegionName and project name.

Updates to python processing for environmental suitability v2.0, as used for operational early warning system.

Updates to perl processing for surveys, deposition and environmental suitability v1.3. Including, as used for operational early warning systems:
* More debug lines
* Call to latest EWS-plotting version