diff --git a/docs/_source/installation.rst b/docs/_source/installation.rst index 90ed47e10b7e915ffbe145b7193033a9285f1620..ebba545d5157af78f0429fef7154343a203620a4 100644 --- a/docs/_source/installation.rst +++ b/docs/_source/installation.rst @@ -28,8 +28,12 @@ In this example, we are using the Linux user “ewsmanager†and group “ews mkdir -p regions/SouthAsia/workspace; mkdir -p regions/EastAfrica/resources; mkdir -p regions/SouthAsia/resources; - mkdir -p met_inputs; - mkdir -p web_data; + mkdir -p met_inputs/metofficeupload/upload/Ethiopia/toMO/; + mkdir -p met_inputs/metofficeupload/upload/Ethiopia/fromMO/daily_name/; + mkdir -p web_data/Ethiopia; + mkdir -p met_inputs/metofficeupload/upload/SouthAsia/toMO/; + mkdir -p met_inputs/metofficeupload/upload/SouthAsia/fromMO/daily_name/; + mkdir -p web_data/SouthAsia; mkdir ./logs; mkdir -p ./envs/credentials @@ -146,7 +150,7 @@ Make a copy of the docker launch template file and edit the source dir to match cd <install_location>/code/coordinator/configs/docker/run cp launchDocker_template.sh launchDocker_<installname>.sh -Launch the docker instance (you will end up inside the docker instance in the “code†dir) +Launch the docker instance (you will end up inside the docker instance in the “code†dir). .. code-block:: bash @@ -156,15 +160,19 @@ Launch the docker instance (you will end up inside the docker instance in the After running the above command, you will have launched the docker image "ews_coordinator" as a container with the name "ews_runner", with your local code and output directories mounted inside the container. You will automatically - be attached to the running container, from which yu can run the commands below. + be attached to the running container, from which you can run the commands below. + + When attached to a Docker container, you can detach, leaving the container running with ctrl+p ctrl+q. To reattach + to a running container run docker attach <container_name>. 5: Run test commands within Docker: ----------------------------------- -You can run all the integration tests, or run “proper†commands that work from met data deployed elsewhere. +You can run all the integration tests within the Docker container (using bundled data), or run commands that connect to +a running Docker container and use met data deployed elsewhere on the filesystem. Run the suite of integration tests: -when inside the running docker container: +when attached to the running docker container: .. code-block:: bash @@ -175,14 +183,16 @@ If successful, you will see lots of system out passing by, with a message like " outputs will be in: /storage/app/EWS_prod/code/coordinator/tests/test_data/test_deployment/regions/EastAfrica/workspace -6: Run a full run ------------------ +6: Run a full run (using docker from outside the container) +----------------------------------------------------------- .. admonition:: Met Office data required This step is to prepare for a full production deployment. You will need to have assess to data from the UK Met Office available on your computer. - Example data can be provided by the Cambridge team. + Example data can be provided by the Cambridge team. **incoming data should be placed into: + <install_location>/met_inputs/metofficeupload/upload/Ethiopia/fromMO/daily_name/ + to run the example commands below.** .. admonition:: Remote accounts required @@ -190,16 +200,24 @@ outputs will be in: /storage/app/EWS_prod/code/coordinator/tests/test_data/test_ information. The Cambridge team will provide you with the necessary credential files, which are referred to in the <install_location>regions/<region>/resources/coordinator/configs/Cred-ODK-EIAR.json file. + **the necessary credentials file will be provided in the resources bundle provided by Cambridge** + + +Edit config files to match your installation +"""""""""""""""""""""""""""""""""""""""""""" + +.. admonition:: Editing not necessary if running in our docker image -Edit config files -""""""""""""""""" + If you are following the steps described above, the default config file will work without needing to be edited. The + Docker container will mount the local paths to match the default config file. You will need to edit the config files + if you are running the pipeline outside of the docker container to match your local filesystem. **Note than when we refer to "the config file" we are refering to the main JSON config that is passed in from the command line when running the pipeline. (e.g.** ``resources/coordinator/configs/config_EastAfrica_fc_live.json``). **each region will have its own config file, e.g. config_EastAfrica_fc_live.json, config_SouthAsia_fc_live.json.** -**Files will be copied using the "scp" command, so the local server will need to be able to run this command -(it is usually included by default with Linux and will run in the Docker container)**. +**Files will be copied using the "scp" command, so your computer will need to be able to run this command if you are not +using Docker (it is usually included by default with Linux and will run by default in the Docker container)**. You will need to edit the config files for each region to point at the correct server paths to retrieve from and push to. @@ -235,20 +253,25 @@ and push to. One all the necessary input data and server connections are in place, you can run the pipeline. Note that if you need to rerun a pipeline, you have to delete the output of the previous run. +.. admonition:: Ensure the Docker container is running + + The following commands run the pipeline by launching a script on a running Docker container. Check the container is + running with "docker container ls", you chould see the ews_runner container running. If it is not running, you can run + it with "docker start ews_runner". + SurveyData ^^^^^^^^^^ (note: day should be the day before the Depo and Env suit dates, also note that the Survey data needs to be run before the Depo and Env suit data) -/storage/app/EWS_prod/code/coordinator/configs/docker/run/runDockerEWS.sh -p Survey -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220807; - +docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Survey_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20230125; Deposition: ^^^^^^^^^^^ -/storage/app/EWS_prod/code/coordinator/configs/docker/run/runDockerEWS.sh -p Deposition -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808 +docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Deposition_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20230126 Env suit: ^^^^^^^^^ -/storage/app/EWS_prod/code/coordinator/configs/docker/run/runDockerEWS.sh -p Environment -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808; +docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Environment_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20230126; EPI: ^^^^ @@ -256,14 +279,15 @@ Set the start of the season be editing the following lines in <install_location> .. code-block:: json - Line 154 "CalculationSpanDays" : ["20220807",5], - Line 225 "seasonStartString" : "20220807", + Line 176 "CalculationSpanDays" : [0,1], + Line 177 "continue" : false, -/storage/app/EWS_prod/code/coordinator/run_Processor.sh -p Epidemiology -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808; +docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Epidemiology_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20230126; -[Currently it requires at least 2 days of output to work properly, throws an error with a single day] +[note that the survey, Environmental Suitability and Deposition pipelines need to be run before the EPI pipeline] Advisory: ^^^^^^^^^ -/storage/app/EWS_prod/code/coordinator/configs/docker/run/runDockerEWS.sh -p Advisory -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808; +docker exec ews_runner /storage/app/EWS_prod/code/coordinator/scripts/run_Advisory_Processor.sh -c /storage/app/EWS_prod/regions/EastAfrica/resources/coordinator/configs/config_EastAfrica_fc_live.json -s 20220808; +[note that the Survey, Environmental Suitability, Deposition and Epidemiology pipelines need to be run before the Advisory pipeline] \ No newline at end of file