From 6324c63d10c685ca4c644f0701197b4765e1f906 Mon Sep 17 00:00:00 2001
From: lb584 <lb584@cam.ac.uk>
Date: Mon, 10 Jul 2023 14:39:13 +0100
Subject: [PATCH] more doc work
---
docs/_source/installation.rst | 97 ++++++++++++++++++++++++++---------
docs/api.rst | 2 +-
2 files changed, 75 insertions(+), 24 deletions(-)
diff --git a/docs/_source/installation.rst b/docs/_source/installation.rst
index 1b262cf..50253f1 100644
--- a/docs/_source/installation.rst
+++ b/docs/_source/installation.rst
@@ -3,9 +3,17 @@ Installation
.. _installation:
-1: Create dirs:
----------------
-If you are planning to have a dedicated user/group for this pipeline, make sure you run these commands using that user
+To use the EWS system in a production environment will require a working relationship with the Cambridge team and the
+UK Met Office. These instructions will allow you to get the EWS system running on your machine, but will require input
+data from the UK Met office (both weather data and the output of NAME runs. The Cambridge team can provide you with the
+required configuration files and project assets for the regions currently being monitored.
+
+The example code below is from a Linux system using the Bash shell.
+
+1: Create installation dirs:
+----------------------------
+If you are planning to have a dedicated user/group for this pipeline, make sure you run these commands using that user.
+In this example, we are using the Linux user “ewsmanager†and group “ewsâ€:
create these dirs as user “ewsmanagerâ€, or chmod them afterwards
@@ -147,42 +155,85 @@ outputs will be in: /storage/app/EWS_prod/code/coordinator/tests/test_data/test_
Example data can be provided by the Cambridge team.
+.. admonition:: Remote accounts required
-SurveyData (note: day should be the day before the Depo and Env suit dates)
+ In addition to the Met Office data, you will need to be able to connect to the various servers which host the survey
+ information. The Cambridge team will provide you with the necessary credential files, which are referred to in the
+ resources/coordinator/configs/Cred-ODK-EIAR.json file.
-rm -v /storage/app/EWS_prod/regions/EastAfrica/workspace/SURVEYDATA_20220501/STATUS_*;/storage/app/EWS_prod/code/coordinator/run_Processor.sh -p Survey -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220807;
-/storage/app/EWS_dev/code/coordinator/run_dev_Processor.sh -p Survey -c /storage/app/EWS_dev/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220807;
+Edit config files
+"""""""""""""""""
+**Note than when we refer to "the config file" we are refering to the main JSON config that is passed in from the command
+line when running the pipeline. (e.g.** ``resources/coordinator/configs/config_EastAfrica_fc_live.json``). **each region will
+have its own config file, e.g. config_EastAfrica_fc_live.json, config_SouthAsia_fc_live.json.**
-Depo:
+**Files will be copied using the "scp" command, so the local server will need to be able to run this command
+(it is usually included by default with Linux)**.
-rm -v /storage/app/EWS_prod/regions/EastAfrica/workspace/DEPOSITION_20220502/STATUS_*;/storage/app/EWS_prod/code/coordinator/run_Processor.sh -p Deposition -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220808
+You will need to edit the config files for each region to point at the correct server paths to retrieve from
+and push to.
-/storage/app/EWS_dev/code/coordinator/run_dev_Processor.sh -p Deposition -c /storage/app/EWS_dev/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220808
+**Set the path to publish pipeline outputs to, e.g.:**
-Env suit:
+.. code-block:: json
-rm -rf /storage/app/EWS_prod/regions/EastAfrica/workspace/ENVIRONMENT_2.0_20220502/; /storage/app/EWS_prod/code/coordinator/run_Processor.sh -p Environment -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220808;
+ "ServerPath" : "/storage/webdir/Ethiopia/",
-rm -rf /storage/app/EWS_dev/regions/EastAfrica/workspace/ENVIRONMENT_2.0_20220808; /storage/app/EWS_dev/code/coordinator/run_dev_Processor.sh -p Environment -c /storage/app/EWS_dev/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220808;
+**For the Environmental suitability pipeline, set the path into which the Met Office is uploading weather and NAME data:**
-EPI
+**For the Deposition pipeline, set the path from which the Met Office downloads the source calculation results, e.g.:**
-Set the start of the season be editing the following lines in <install_location>/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json
+.. code-block:: json
-Line 154 "CalculationSpanDays" : ["20220807",5],
-Line 225 "seasonStartString" : "20220807",
+ "Environment" : {
+ "ServerPathTemplate" : "/storage/sftp/metofficeupload/fromMO/Ethiopia/daily_name/",
+ "Deposition" : {
+ "ServerPathTemplate" : "/storage/sftp/metofficeupload/toMO/Ethiopia/daily_name/",
-rm /storage/app/EWS_prod/regions/EastAfrica/workspace/EPI_20220502/STATUS_*; /storage/app/EWS_prod/code/coordinator/run_Processor.sh -p Epidemiology -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220808;
+**Specify the putput directory of the Survey pipeline, which should be the location from which the Met Office can download the results in order to feed them into the NAME calculation, e.g.**
-[Currently it requires at least 2 days of output to work properly, throws an error with a single day]
-Advisory
+.. code-block:: json
+
+ "Survey" : {
+ "ServerPathExtra" : "/storage/sftp/metofficeupload/upload/Ethiopia/toMO/",
+
+**Running the pipeline**
+
+One all the necessary input data and server connections are in place, you can run the pipeline. Note that if you need to
+rerun a pipeline, you have to delete the output of the previous run.
+
+SurveyData
+^^^^^^^^^^
+(note: day should be the day before the Depo and Env suit dates, also note that the Survey data needs to be run before the Depo and Env suit data)
+
+/storage/app/EWS_prod/code/coordinator/configs/docker/run/runDockerEWS.sh -p Survey -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220807;
-rm /storage/app/EWS_prod/regions/EastAfrica/workspace/SUMMARY_20220502/STATUS_*;
-/storage/app/EWS_prod/code/coordinator/run_Processor.sh -p Advisory -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220808;
-rm /storage/app/EWS_dev/regions/EastAfrica/workspace/SUMMARY_20220502/STATUS_*;
-/storage/app/EWS_dev/code/coordinator/run_dev_Processor.sh -p Advisory -c /storage/app/EWS_dev/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json --noupload -s 20220808;
+Deposition:
+^^^^^^^^^^^
+/storage/app/EWS_prod/code/coordinator/configs/docker/run/runDockerEWS.sh -p Deposition -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808
+
+Env suit:
+^^^^^^^^^
+/storage/app/EWS_prod/code/coordinator/configs/docker/run/runDockerEWS.sh -p Environment -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808;
+
+EPI:
+^^^^
+Set the start of the season be editing the following lines in <install_location>/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json
+
+.. code-block:: json
+
+ Line 154 "CalculationSpanDays" : ["20220807",5],
+ Line 225 "seasonStartString" : "20220807",
+
+/storage/app/EWS_prod/code/coordinator/run_Processor.sh -p Epidemiology -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808;
+
+[Currently it requires at least 2 days of output to work properly, throws an error with a single day]
+
+Advisory:
+^^^^^^^^
+/storage/app/EWS_prod/code/coordinator/configs/docker/run/runDockerEWS.sh -p Advisory -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808;
diff --git a/docs/api.rst b/docs/api.rst
index 92c6af1..3d57073 100644
--- a/docs/api.rst
+++ b/docs/api.rst
@@ -6,7 +6,7 @@ API
:template: custom-module-template.rst
:recursive:
- coordinator
+.. coordinator
plotting
ews_postprocessing
met_processing
--
GitLab