installation.rst

Installation

To use the EWS system in a production environment will require a working relationship with the Cambridge team and the UK Met Office. These instructions will allow you to get the EWS system running on your machine, but will require input data from the UK Met office (both weather data and the output of NAME runs. The Cambridge team can provide you with the required configuration files and project assets for the regions currently being monitored.

See also :ref:`deployment notes <deployment_notes>`

The example code below is from a Linux system using the Bash shell. The pipeline runs within a Docker container (running Linux). If you are using Windows, you will need to run the initial directory creation and Git commands using Windows equivalents.

THESE INSTRUCTIONS ASSUME YOU HAVE ADMIN PRIVILEGES ON THE COMPUTER YOU ARE USING, AND ARE FAMILIAR WITH DOCKER, If you are not familiar with Docker, you should be able to follow the instructions, but may run into difficulties you go off-track.

1: Create installation dirs:

If you are planning to have a dedicated user/group for this pipeline, make sure you run these commands using that user. In this example, we are using the Linux user “ewsmanager” and group “ews”. It is fine to use the default user/group.

cd <install_location>
mkdir ./code;
mkdir -p regions/EastAfrica/workspace;
mkdir -p regions/SouthAsia/workspace;
mkdir -p regions/EastAfrica/resources;
mkdir -p regions/SouthAsia/resources;
mkdir -p met_inputs/metofficeupload/upload/Ethiopia/toMO/;
mkdir -p met_inputs/metofficeupload/upload/Ethiopia/fromMO/daily_name/;
mkdir -p web_data/Ethiopia;
mkdir -p met_inputs/metofficeupload/upload/SouthAsia/toMO/;
mkdir -p met_inputs/metofficeupload/upload/SouthAsia/fromMO/daily_name/;
mkdir -p web_data/SouthAsia;
mkdir ./logs;
mkdir -p ./envs/credentials

mkdir -p ./code regions/EastAfrica/workspace regions/SouthAsia/workspace regions/EastAfrica/resources regions/SouthAsia/resources met_inputs/metofficeupload/upload/Ethiopia/toMO/  met_inputs/metofficeupload/upload/Ethiopia/fromMO/daily_name/ web_data/Ethiopia met_inputs/metofficeupload/upload/SouthAsia/toMO/ met_inputs/metofficeupload/upload/SouthAsia/fromMO/daily_name/ web_data/SouthAsia ./logs ./envs/credentials

mkdir -p .\code regions\EastAfrica\workspace regions\SouthAsia\workspace regions\EastAfrica\resources regions\SouthAsia\resources met_inputs\metofficeupload\upload\Ethiopia\toMO\  met_inputs\metofficeupload\upload\Ethiopia\fromMO\daily_name\ web_data\Ethiopia met_inputs\metofficeupload\upload\SouthAsia\toMO\ met_inputs\metofficeupload\upload\SouthAsia\fromMO\daily_name\ web_data\SouthAsia .\logs .\envs\credentials

shell scripts to perform the above operations: :download:`Download .cmd file <../_static/assets/windows_create_dirs.cmd>` :download:`Download .sh file <../_static/assets/linux_create_dirs.sh>`

chmod -R g+sw <install_location>

make all files written here default to the current user group, and group-writable. This is not necessary if you are depoying in a single-user environment, such as a personal computer

2: Clone code

(can copy-paste below as a load of commands to run on one line) Assumes you have permissions on our gitlab repo:

cd into <install_location>/code

git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-coordinator.git coordinator;git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-environmental-suitability.git environmental_suitability;git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-advisory-builder.git advisory_builder;git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-plotting.git plotting;git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-epidemiology.git epimodel;git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-source-calculation.git source_gen;git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-job-flagging.git flagdir;git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-met-processing.git met_extractor_v2;git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/ews-postprocessing.git post_processing;

or generate this string in bash:

for package in ews-coordinator,coordinator \
               ews-environmental-suitability,environmental_suitability \
               ews-advisory-builder,advisory_builder \
               ews-plotting,plotting \
               ews-epidemiology,epimodel \
               ews-source-calculation,source_gen \
               ews-job-flagging,flagdir \
               ews-met-processing,met_extractor_v2 \
               ews-postprocessing,post_processing;
do
  repo=`echo $package | cut -d "," -f 1`
  folder=`echo $package | cut -d "," -f 2`
  printf "%s" "git clone git@gitlab.developers.cam.ac.uk:gilligan-epid/wheat-rusts/${repo}.git ${folder};"
done

shell scripts to perform the above operations: :download:`Download .cmd file <../_static/assets/windows_clone_git.cmd>` :download:`Download .sh file <../_static/assets/linux_clone_git.sh>`

3: Copy the Resources needed for the various chart areas:

Configuration files from Cambridge required

Due to the size, project resources are not included in the git repository. You will need to ask the Cambridge team for the latest version of the resources archive.

Add resources for one region at a time

There is a seperate resources archive for each geographic region. Currently there are two regions, EastAfrica and SouthAsia. The instructions below denote the different regions with <EastAfrica/SouthAsia>. We suggest you start with EastAfrica and get that working before setting up other regions.

Once you have received an archive of <EastAfrica/SouthAsia>_resources_<latest_date>.tar.gz from Cambridge, unpack into <install_location>/regions/<EastAfrica/SouthAsia>/resources. This will place all the necessary JSON configuration and other assets required for the pipeline to run.

e.g.:

cd <install_location>/regions/EastAfrica/resources
tar -zxvf eastafrica_resources_<latest_date>.tar.gz

Edit the coordinator run script to fix the root paths (not necessary if running in our docker image):

<install_location>/code/coordinator/scripts/run_Processor.py

Put a valid email credentials file into <install_location>/envs/credentials/Cred_gmail.json

Example format:

{
    "user" : "clusternotifications@gmail.com",
    "pass" : "passwordhere",
    "host" : "smtp.gmail.com",
    "port" : 587,
    "toaddrs" : ["person1@cam.ac.uk","person1@cam.ac.uk"]
}

Specify your own email server

The example credentials above are for the Cambridge email account. You will need to replace the host/user etc with your own.

4: Run the docker image:

install docker on your machine https://docs.docker.com/get-docker/

Build the docker image (slow, one-off step)

cd <install_location>/code/coordinator/configs/docker/build/

By default, the buildfile will create an image with the user credentials of the Cam Epi group server. To make a version with the right credentials to write you your filesystem, run the build script provided, but edit to override the following credentials:

(from build.sh)

sudo docker build -t ews_coordinator --build-arg GNAME=<your value> --build-arg UNAME=<your value>  --build-arg UID=<your value> --build-arg GID=<your value> -f Dockerfile .

Where the various variables are all LINUX user credentials (type the ‘id’ command to see all these values)

if you are running the ews for use on your personal computer, you can exclude the --build-arg arguments from the build step, note that all files written by the pipeline will be owned by the root user, rather than your user account.

Once you have edited the build.sh script, run it (or run the commands within it manually). **if you are running on Windows and run these commands manually, make sure you run the "cp ../../conda/conda-env-py3EWS-withbuilds.yml ." equivalent to place the conda dependencies in the same dir before running docker build.

bash build.sh

building docker can take a while, typically 5-10 mins

Make a copy of the docker launch template file and edit the source dir to match that of your installation.

cd <install_location>/code/coordinator/configs/docker/run
cp launchDocker_template.sh launchDocker_<installname>.sh

Launch the docker instance (you will end up inside the docker instance in the “code” dir).

bash launchDocker_<installname>.sh

What launchDocker_<installname>.sh does

After running the above command, you will have launched the docker image "ews_coordinator" as a container with the name "ews_runner", with your local code and output directories mounted inside the container. You will automatically be attached to the running container, from which you can run the commands below.

When attached to a Docker container, you can detach, leaving the container running with ctrl+p+q. To reattach to a running container run docker attach <container_name>.

5: Run test commands within Docker:

You can run all the integration tests within the Docker container (using bundled data), or run commands that connect to a running Docker container and use met data deployed elsewhere on the filesystem.

Run the suite of integration tests:

when attached to the running docker container:

cd /storage/app/EWS_prod/code/coordinator/tests/integration/partial/
./run_test_suite.sh

If successful, you will see lots of system out passing by, with a message like "Ran 14 tests in 0:01:13 OK" at the end.

outputs will be in: /storage/app/EWS_prod/code/coordinator/tests/test_data/test_deployment/regions/EastAfrica/workspace

6: Run a full run (using docker from outside the container)

Met Office data required

This step is to prepare for a full production deployment. You will need to have access to data from the UK Met Office available on your computer.

Example data can be provided by the Cambridge team. incoming data should be placed into: <install_location>/met_inputs/metofficeupload/upload/Ethiopia/fromMO/daily_name/ to run the example commands below.

Remote accounts required

In addition to the Met Office data, you will need to be able to connect to the various servers which host the survey information. The Cambridge team will provide you with the necessary credential files, which are referred to in the <install_location>regions/<region>/resources/coordinator/configs/Cred-ODK-EIAR.json file.

the necessary credentials file will be provided in the resources bundle provided by Cambridge

Edit config files to match your installation

Editing not necessary if running in our docker image

If you are following the steps described above, the default config file will work without needing to be edited. The Docker container will mount the local paths to match the default config file. You will need to edit the config files if you are running the pipeline outside of the docker container to match your local filesystem.

Note than when we refer to "the config file" we are refering to the main JSON config that is passed in from the command line when running the pipeline. (e.g. resources/coordinator/configs/config_EastAfrica_fc_live.json). each region will have its own config file, e.g. config_EastAfrica_fc_live.json, config_SouthAsia_fc_live.json.

Files will be copied using the "scp" command, so your computer will need to be able to run this command if you are not using Docker (it is usually included by default with Linux and will run by default in the Docker container).

You will need to edit the config files for each region to point at the correct server paths to retrieve from and push to.

Set the path to publish pipeline outputs to, e.g.:

"ServerPath" : "/storage/webdir/Ethiopia/",

For the Environmental suitability pipeline, set the path into which the Met Office is uploading weather and NAME data:

For the Deposition pipeline, set the path from which the Met Office downloads the source calculation results, e.g.:

"Environment" : {
    "ServerPathTemplate" : "/storage/sftp/metofficeupload/fromMO/Ethiopia/daily_name/",
"Deposition" : {
    "ServerPathTemplate" : "/storage/sftp/metofficeupload/toMO/Ethiopia/daily_name/",

Specify the output directory of the Survey pipeline, which should be the location from which the Met Office can download the results in order to feed them into the NAME calculation, e.g.

"Survey" : {
    "ServerPathExtra" : "/storage/sftp/metofficeupload/upload/Ethiopia/toMO/",

Running the pipeline

One all the necessary input data and server connections are in place, you can run the pipeline. Note that if you need to rerun a pipeline, you have to delete the output of the previous run.

Ensure the Docker container is running

The following commands run the pipeline by launching a script on a running Docker container. Check the container is running with "docker container ls -a", you chould see the "ews_runner" container running. If it is not running, you can run it with "docker start ews_runner". Note that the "ews_runner" container will exist because you have created the container while running the "launchDocker_<installname>.sh" script described above.

SurveyData

(note: day should be the day before the Depo and Env suit dates, also note that the Survey data needs to be run before the Depo and Env suit data)

docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Survey_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20230125;

Deposition:

docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Deposition_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20230126

Env suit:

docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Environment_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20230126;

EPI:

Set the start of the season be editing the following lines in <install_location>/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json

Line 176 "CalculationSpanDays" : [0,1],
Line 177 "continue" : false,

docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Epidemiology_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20230126;

[note that the survey, Environmental Suitability and Deposition pipelines need to be run before the EPI pipeline]

Advisory:

docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Advisory_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808;

[note that the Survey, Environmental Suitability, Deposition and Epidemiology pipelines need to be run before the Advisory pipeline]