A basic script that checks the ARRCC media scraper web tool for latest
entries and provides rough formatting for input to NAME source calcs.
The idea is to email this to a list of maintainers (me, @tm689, @rs481)
regularly (once a week). Because the system isn't smart enough to filter
out any irrelevant news reports (e.g. general early season advice
without a specific rust sighting), human intervention is required to
filter and append to the survey data. The media scraper checker is set
as a cron task to run every Monday at midday.

Back run handling. If env.suit. results are available in a _bak folder, copy results from the _bak folder and skip env.suit. calc.

The method in this example may become the official method of downoading Ethiopia wheat rust surveys. Based on proposed API provided by CSM on 23 September 2020.

A useful function in ProcessorUtils is subprocess_and_log, which builds a subprocess command and write its stdout and stderr to the log handler. Beforehand, failures only ever generated an error message, but there are some use cases where another log type is preferred. This adds a log type option to the function, and uses it in one case where a warning is more appropriate.

From 3rd February 2021 each Wednesday, Will at the Met Office will send deposition and env suit met calcs using analysis meteorology. The results can be compared with standard calcs using forecast meteorology. To get a simple reproduction of the layout, the analysis jobs are handled in a separate workspace. Crontab will be switched on after 27th Jan.

The tarred directory of files from the Met Office includes the NAME .txt files to use as input, as well as various log files and diagnostic plots. The number of log files changed recently but shouldn't affect the operatinal calcs. This change means the check for a complete transfer of incoming files cares only about the NAME .txt files.

Output was tested against EWS-Coordinator v1.1 and found to be the same. v1.1 and v2.0 were running alongside for two weeks to monitor the new system and it performed as well as v1.1.

This should only apply to the operational copy of the EWS-Coordinator. If you want to run a separate copy for any reason, make these changes to point to your separate workspace. The git ignored files still need pointing to, but they don't need to be duplicated. Your local version of these changes can be stashed while pushing and pulling to/from the master branch.

In operational version, this is needed for survey results. The way this is implemented, it can be configured for any other component too.

Updating Env Suit v2.0 calculation. Copied from operational EnvProcessor.py in master branch. Not tested for this commit.

Hard-coded check of length of exported survey data as a basic check of ODK Briefcase for robustness (in particular for exports from Ethiopia). Currently, 2715 is an expected minimum for Ethiopia. Any small increases in coming weeks are unlikely to lead to not catching an incomplete export. This check is not important for SouthAsia data export which has been reliable. Because of the large data size, this 2715 limit is almost ineffective for South Asia anyway.

Errors in process_pre_job can create a job directory to record the error (instead of rediscovering every time the script is called).