IaaS setup
==========

.. _iaas_setup:

This isnt included in the main docs, only linkable with a direct URL

create new Ubuntu instance using the IaaS broker service:

https://v-vra01.srv.uis.private.cam.ac.uk/catalog/#

latest instance is:

ssh lb584@10.136.11.3
BOT-cags1

mount storage
-------------

mount our RDS and RFS storage

the mounting of RDS and RFS is a workaround until we can purchase block storage on the IaaS. it works, but does not preserve different user permissions and is slow.

sudo apt-get install cifs-utils
sudo apt install smbclient

create dirs to mount:
sudo mkdir /mnt/RDS
sudo mkdir /mnt/RFS

add to /etc/fstab

//hpc-isi-w.hpc.private.cam.ac.uk/rfs-x2jwbL2P9T4	/mnt/RFS	cifs	credentials=/root/.rfs_credentials,workgroup=BLUE.CAM.AC.UK,file_mode=0775,dir_mode=0775,uid=1006,gid=1008	0	0
lb584@rds.uis.cam.ac.uk:rds-ews_production-rjVtFkyQ1T0/	/mnt/RDS/	fuse.sshfs	defaults,noauto,_netdev,allow_other,uid=1006,gid=1008	0	0

notes: uid and gid are for the ewsmanager user, which has not actually been made yet, and will have a different id on a new install

the sshfs mount will not get mounted automatically (has the “noauto” option) and will require this to mount:

sudo mount /mnt/RDS - and will prompt for the password. note that this is being mounted as lb584, a different owner will need to supply different credentials

the credentials stored for the RFS in /root/.rfs_credentials are also going to need to be adjusted to the appropriate user is lb584 is no longer about. It doesn't matter who mounts them (provided they have access to the storage set by the datamanager of the storage) as they both have the uid set to be the shared ewsmanager user.

run sudo dpkg-reconfigure tzdata to make sure it automatically adjusts the clock for BST/GMT
Basic user and groups setup

1: install needed packages
--------------------------

initial login must be as root

apt-get update
apt-get install postfix
apt-get install emacs

open emacs and set the theme:
alt-x customize-themes

2: create users
---------------

sudo adduser lb584,jws52,tm689,ewsmanager

3: add ews group
----------------

sudo groupadd ews

4: add users to the ews group
-----------------------------

sudo usermod -aG ews ewsmanager/jwc52/tm689

5: allow members of ews group to su into ewsmanager user:
---------------------------------------------------------

sudo emacs -nw /etc/pam.d/su

# This allows root to su without passwords (normal operation)
auth       sufficient pam_rootok.so

auth  [success=ignore default=1] pam_succeed_if.so user = ewsmanager
auth  sufficient                 pam_succeed_if.so use_uid user ingroup ews

6: grant root users sudo privileges
-----------------------------------

usermod -aG sudo lb584

7: Install docker
-----------------

download .deb from https://docs.docker.com/desktop/install/ubuntu/

or

wget -O docker-desktop-4.14.1-amd64.deb2 https://desktop.docker.com/linux/main/amd64/docker-desktop-4.14.1-amd64.deb?utm_source=docker&utm_medium=webreferral&utm_campaign=docs-driven-download-linux-amd64

install docker engine: (instructions at https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository)

sudo apt-get install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null


sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

**add docker users:**
sudo groupadd docker
sudo usermod -aG docker $USER (reconnect/reopen terminal)

**check docker works:**

<sudo> docker run hello-world (wont need sudo if the user is added to the docker group)

Add sftp access for metoffice
-----------------------------

groupadd sftponly
${EDITOR:-nano} /etc/ssh/sshd_config

Match Group sftponly
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no
ChrootDirectory /storage/sftp/%u
ForceCommand internal-sftp

service ssh start
mkdir -p /storage/sftp/
chown -R root:root /storage/sftp/
chmod -R 755 /storage/sftp/

###REPEAT THIS PROCESS FOR THE PLANTVILAGE USER####
adduser --no-create-home --shell /usr/sbin/nologin metofficeupload   # create the user and an associated group with the same name
adduser metofficeupload sftponly   # add to existing sftponly group to allow (only) the sftp subsystem to be used under the chroot
mkdir -p /storage/sftp/metofficeupload/upload

chown root:root /storage/sftp/metofficeupload
chown metofficeupload:metofficeupload /storage/sftp/metofficeupload/upload
chmod -R 755 /storage/sftp/metofficeupload
mkdir -p /storage/sftp/metofficeupload/upload/Ethiopia/fromMO/daily_name
mkdir -p /storage/sftp/metofficeupload/upload/Ethiopia/toMO/

sftp metofficeupload@10.136.11.3

add the ewsmanager user to metofficeupload group so it can manage files written by this user

sudo usermod -aG metofficeupload ewsmanager
###END REPEAT THIS PROCESS FOR THE PLANTVILAGE USER####


Deploy and test the EWS code
----------------------------

**1: add the ssh key of the server to gitlab for access.**

generate ssh key (if dont have one already)
ssh-keygen -t rsa -b 4096 -C <key_id>
https://gitlab.developers.cam.ac.uk/-/profile/keys

**2: make dirs for the EWS app**

mkdir /storage/app/EWS_prod
sudo chown -R ewsmanager:ews EWS_prod/
sudo chmod -R g+s EWS_prod/
sudo chmod -R g+w EWS_prod/

**3: follow the deployment instructions in this doc.**

https://docs.google.com/document/d/1nW0eZJoLLOFzb3Yp4OonhwrbOS2UFELok_QQflb-EUc/edit?usp=sharing
Install and setup the apache server (file downloads and ews_browser)

setup dirs for apache:

sudo mkdir /storage/webdata/Ethiopia
sudo mkdir /storage/webdata/SouthAsia
sudo chown ewsmanager:ews SouthAsia/ Ethiopia/
sudo chmod g+ws Ethiopia/
sudo chmod g+ws SouthAsia/

sudo mkdir /storage/app/ews_browser
sudo chown ewsmanager:ews ews_browser
sudo chmod g+ws ews_browser

ln -s /storage/webdata/Ethiopia /var/www/html/Ethiopia
ln -s /storage/webdata/SouthAsia /var/www/html/SouthAsia

install apache and other libs:

apt-get -qq install --assume-yes apache2 apache2-dev apache2-utils ssl-cert libapache2-mod-wsgi openssh-server

(if you get prompted about a newer version of the sshd_conf file being available, keep the current one as we modified this already.)

install miniconda:

export CONDA_DIR='/home/miniconda3'

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /home/miniconda.sh
/bin/bash /home/miniconda.sh -b -p $CONDA_DIR

export PATH=$CONDA_DIR/bin:$PATH
conda init bash

create a file called ews_browser.yml and paste the following:

name:  /home/conda/ews_browser
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - ca-certificates=2022.6.15=ha878542_0
  - click=8.1.3=py38h578d9bd_0
  - flask=2.2.2=pyhd8ed1ab_0
  - importlib-metadata=4.11.4=py38h578d9bd_0
  - itsdangerous=2.1.2=pyhd8ed1ab_0
  - jinja2=3.1.2=pyhd8ed1ab_1
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=12.1.0=h8d9b700_16
  - libgomp=12.1.0=h8d9b700_16
  - libnsl=2.0.0=h7f98852_0
  - libsqlite=3.39.2=h753d276_1
  - libstdcxx-ng=12.1.0=ha89aaad_16
  - libzlib=1.2.12=h166bdaf_2
  - markupsafe=2.1.1=py38h0a891b7_1
  - ncurses=6.3=h27087fc_1
  - openssl=3.0.5=h166bdaf_1
  - pip=22.2.2=pyhd8ed1ab_0
  - python=3.8.12=h0744224_3_cpython
  - python_abi=3.8=2_cp38
  - readline=8.1.2=h0f457ee_0
  - setuptools=65.3.0=pyhd8ed1ab_1
  - sqlite=3.39.2=h4ff8645_1
  - tk=8.6.12=h27826a3_0
  - werkzeug=2.2.2=pyhd8ed1ab_0
  - wheel=0.37.1=pyhd8ed1ab_0
  - xz=5.2.6=h166bdaf_0
  - zipp=3.8.1=pyhd8ed1ab_0
  - pip:
    - mod-wsgi==4.9.3
prefix: /home/conda/ews_browser

conda env create -f /home/ews_browser.yml -p /home/conda_envs/ews_browser

(faff) modify the wsgi.load script in /etc/apache2/mods-available to point to the same python as used by the ews_browser

Enable the conda env for the project:

conda activate /home/conda_envs/ews_browser

which mod_wsgi-express

gives you:

/home/conda_envs/ews_browser/bin/mod_wsgi-express

sudo /home/conda_envs/ews_browser/bin/mod_wsgi-express install-module

gives you:

LoadModule wsgi_module "/usr/lib/apache2/modules/mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so"
WSGIPythonHome "/home/conda_envs/ews_browser"

put the above line as the text in:

/etc/apache2/mods-available/wsgi.load (replacing what is currently there)

This means that apache wsgi will use the version of python that is used by the app to which it is binding.

prepare certificates and passwords for apache:

copy the default-ssl.conf file into /etc/apache2/sites_available. See notes on installing genuine certificates below (once the server is up and running).

<IfModule mod_ssl.c>
	<VirtualHost _default_:443>
		ServerAdmin webmaster@localhost

		DocumentRoot /var/www/html


		ErrorLog ${APACHE_LOG_DIR}/error.log
		CustomLog ${APACHE_LOG_DIR}/access.log combined
		SSLEngine on

		SSLCertificateFile	/etc/apache2/ssl/server.crt
		SSLCertificateKeyFile /etc/apache2/ssl/server.key

		<FilesMatch "\.(cgi|shtml|phtml|php)$">
				SSLOptions +StdEnvVars
		</FilesMatch>
		<Directory /usr/lib/cgi-bin>
				SSLOptions +StdEnvVars
		</Directory>


    <Directory "/var/www/html/Ethiopia">
       AuthType Basic
       AuthName "Restricted Content"
       AuthUserFile /etc/apache2/.htpasswd
       Require user ethiopia
       Options +Indexes +FollowSymLinks +Includes +MultiViews
    </Directory>
    <Directory "/var/www/html/SouthAsia">
       AuthType Basic
       AuthName "Restricted Content"
       AuthUserFile /etc/apache2/.htpasswd
       Require user southasia
       Options +Indexes +FollowSymLinks +Includes +MultiViews
    </Directory>

    TypesConfig /etc/mime.types

	</VirtualHost>
</IfModule>

enable url rewriting in apache:
-------------------------------

This allows a maintenance page to be displayed if services are down.

a2enmod rewrite;

edit the apache conf at /etc/apache2/apache2.conf

<Directory /var/www/>
	Options Indexes FollowSymLinks
	AllowOverride All
	Require all granted
</Directory>

add a maintenance.html file to /var/www/html e.g.

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <STYLE type="text/css">
      body {font-family:sans-serif; color: black; background: white;}
      div {text-align: center}
    </STYLE>
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
    <title>Willow server homepage</title>
  </head>
  <body>
    <center>
      <br><br>
      <div><i>This service is now running at <a href="https://epi.plantsci.cam.ac.uk/">https://epi.plantsci.cam.ac.uk</a> - please edit your url and bookmarks accordingly.</i></div>
    </center>
  </body>
</html>


**add an .htaccess file in /var/www/html/.htaccess**

ErrorDocument 503 /maintenance.html
RewriteEngine On
RewriteCond %{REMOTE_ADDR} !^000\.000\.000\.000
RewriteCond %{REQUEST_URI} !/maintenance.html$ [NC]
RewriteRule .* - [L,R=503]

The RewriteEngine is set to “Off” when not redirecting

**modify the ews browser wsgi conf files to point to the html directory, rather than the code:**

/etc/apache2/sites-available/ews_browser_client_africa.conf
/etc/apache2/sites-available/ews_browser_client_asia.conf

**comment in the directory mapping in place of the  WSGI mapping. e.g.**

WSGIDaemonProcess ews_browser_asia user=ewsmanager group=ewsmanager threads=5 python-home=/home/conda_envs/ews_browser/
 WSGIScriptAlias /ews_browser_asia /storage/app/ews_browser/code/src/main/python/ews_browser_asia.wsgi
 <Directory "/storage/app/ews_browser/code/src/main/python/">
 	   WSGIProcessGroup ews_browser_asia
 	   WSGIScriptReloading On
 	   WSGIApplicationGroup %{RESOURCE}
 	   Require all granted
 </Directory>
 ErrorLog /storage/app/ews_browser/outputs/ews_browser_client_error.log

# IF THE SITE IS DOWN FOR MAINTENANCE, COMMENT THIS BLOCK IN AND THE ONE ABOVE OUT - IT WILL USE THE REDIRECT IN .HTACCESS TO DISPLAY THE MAINTENANCE MESSAGE
#<Directory /var/www/>
#        Options Indexes FollowSymLinks
#        AllowOverride All
#        Require all granted
#</Directory>


**restart the server to implement the changes:**

sudo service apache2 restart

This will make all traffic to the /var/www/html dir redirect to the maintenance page.

enable ssl in apache:
---------------------

ln -s /etc/apache2/sites-available/default-ssl.conf /etc/apache2/sites-enabled/000-default-ssl.conf

a2enmod ssl (possible this is already enabled)

**add passwords for Ethiopia and SouthAsia users:**

touch /etc/apache2/.htpasswd

(get the password from someone or make a new one)
htpasswd /etc/apache2/.htpasswd southasia
htpasswd /etc/apache2/.htpasswd ethiopia

**create server certificates - once you have a domain name registered and pointing at this instance:**

sudo snap install --classic certbot (one off, to install software)

sudo ln -s /snap/bin/certbot /usr/bin/certbot

**then generate the certificates using:**

sudo certbot certonly --apache

(enter the domain name you want to register (which needs to be wired through to this instance)

note that this retrieves the certificate only, does not install them

This created cert files in:

/etc/letsencrypt/live/epi.plantsci.cam.ac.uk (where epi.plantsci.cam.ac.uk is the registered domain (in this example))

I then edited the certificate path lines in this file:

/etc/apache2/sites-enabled/ssl.conf

to:

SSLCertificateFile /etc/letsencrypt/live/epi.plantsci.cam.ac.uk/fullchain.pem

SSLCertificateKeyFile /etc/letsencrypt/live/epi.plantsci.cam.ac.uk/privkey.pem

These certificates should be auto-renewed as certbot sets up a timer task to check if renewal is needed. See the timer by running:

systemctl list-timers

make the output dirs for the pipeline:
-------------------------------------

mkdir -p /storage/app/ews_browser/outputs/temp_unzip_dir
chmod -R 775 /storage/app/ews_browser/outputs/
chmod g+sw /storage/app/ews_browser/outputs/

deploy the ews_browser code
---------------------------

**copy the ews_browser code into /storage/app/ews_browser**

**create a symlink to the wsgi conf files**

ln -s /storage/app/ews_browser/code/src/main/python/ews_browser_client_asia.conf ews_browser_client_asia.conf
ln -s /storage/app/ews_browser/code/src/main/python/ews_browser_client_africa.conf ews_browser_client_africa.conf

**activate the sites:**

a2ensite ews_browser_client_africa.conf
a2ensite ews_browser_client_asia.conf

**start server:**

service apache2 start

**setup gitlab runner for CI**

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

note: on the IaaS I had to “mkdir /var/lib/gitlab-runner” as the logs were complaining and the service wasnt staying up. I'm hoping this was because I upgraded the runner version and something got out of whack.

sudo gitlab-runner register
go into /etc/gitlab-runner as root and edit the newly registered runner in config.toml :

concurrent = 1
check_interval = 0

[[runners]]
  name = "bot-cags1-production"
  url = "https://gitlab.developers.cam.ac.uk/"
  token = "sjgyL15rgdC1veHs6yPT"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "python"
    privileged = false
    disable_cache = false
    volumes = ["/storage/webdata:/storage/moved","/storage/sftp:/storage/sftp","/storage/app/EWS_prod/envs/credentials:/storage/app/EWS_prod/envs/credentials","/storage/app/EWS_prod/regions:/storage/app/EWS_prod/regions","/storage/app/EWS_prod/code:/storage/app/EWS_prod/code","/cache"]
    shm_size = 0
  [runners.cache]

Airflow installation
====================

Install mysql:

https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-20-04

sudo apt install mysql-server

sudo systemctl start mysql.service

hack to allow root access via sudo on ubuntu (in the digital ocean docs)

sudo mysql
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password_here';
exit

sudo mysql_secure_installation
(no opinion on what options are selected, except disabling root access from remote host (yes))

finish the hack for ubuntu root access:
mysql -u root -p
ALTER USER 'root'@'localhost' IDENTIFIED WITH auth_socket;
exit
</end ubuntu hack>

Install Airflow
---------------

https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

mkdir <path>/airflow
cd airflow
set the airflow home location
export AIRFLOW_HOME=<path>/airflow

python3 -m venv airflow-env;
source airflow-env/bin/activate;

pip install wheel;
pip install apache-airflow[celery]==2.7.1 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.8.txt"

note that there are two dashes before constraint above, not a single long dash

pip check

**install the docker operator**

pip install apache-airflow-providers-docker

**install mysql (need root privileges)**

https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-20-04

sudo apt update;
sudo apt install mysql-server;
sudo systemctl start mysql.service;

follow the additional instructions to work around the mysql_secure_installation issue

sudo mysql;
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '<put_root_password_here>';
exit;

The password above has been set to the same as the ewsmanager password. You can now run the security manager script:

sudo mysql_secure_installation;

<choose sensible security options at your discretion>

**Setup mysql database:**

mysql -u root -p

CREATE DATABASE airflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

CREATE USER 'airflow_user'@'localhost' IDENTIFIED BY 'generate_a_non_secret_password_for_here';

GRANT ALL PRIVILEGES ON airflow_db.* TO 'airflow_user';

exit;


**set database defaults (sqllite)**

airflow db migrate

edit the resulting ${AIRFLOW_HOME}/airflow.cfg file on line 435 to the mysyl database
sql_alchemy_conn = mysql+mysqldb://airflow_user:<password_from_above>@localhost:3306/airflow_db

while you are editing the config file, set the timezone to be your local timezone, rather than UTC. As this system works
on a regular, local timezone, daylight saving hours need to be taken into account so it runs at the same local time each day.

default_timezone = Europe/London
(note that the UI has a dropdown in the top right to set the displayed timezone, but the scheduler will run on the server timezone)

**install the mysql provider**

sudo apt install gcc;
sudo apt-get install python3-dev;
pip install apache-airflow-providers-mysql

the above may fail if you dont specify the mysql headers first as in this post:

https://stackoverflow.com/questions/76875507/can-not-install-apache-airflow-providers-mysql-pkg-config-error

sudo apt install libmysqlclient-dev;
export MYSQLCLIENT_CFLAGS="$(mysql_config --cflags)";
export MYSQLCLIENT_LDFLAGS="$(mysql_config --libs)";

**migrate the db again to use mysql this time**

airflow db migrate

possible sql mode error:

https://stackoverflow.com/questions/36882149/error-1067-42000-invalid-default-value-for-created-at

create an admin user for the webserver

airflow users  create --role Admin --username admin --email lb584@cam.ac.uk --firstname admin --lastname admin --password admin

(you can set the password to something more secure when logged in)



run the scheduler and the webserver

airflow scheduler;
airflow webserver;

can run as demons with the -D flag

connect to the webserver http://<server_ip>:8090

**set up airflow as a service**

Once you have got airflow running from the command line, you will need to set it up as a service (which will run at startup and in the background)

1: create a service file for airflow scheduler and webserver:

sudo touch /etc/systemd/system/airflow-scheduler.service
sudo touch /etc/systemd/system/airflow-webserver.service

edit the contents of both files to be:

.. code-block:: bash

    [Unit]
    Description=Airflow scheduler daemon
    After=network.target mysql.service
    Wants=mysql.service
    [Service]
    EnvironmentFile=/storage/airflow/airflow.cfg
    User=ewsmanager
    Group=ewsmanager
    Type=simple
    ExecStart=/usr/bin/bash -c 'export AIRFLOW_HOME=/storage/airflow ; source  /storage/airflow/airflow-env/bin/activate ; airflow webserver'
    Restart=no
    RestartSec=5s
    PrivateTmp=true
    [Install]
    WantedBy=multi-user.target

.. code-block:: bash

    [Unit]
    Description=Airflow scheduler daemon
    After=network.target mysql.service
    Wants=mysql.service
    [Service]
    EnvironmentFile=/storage/airflow/airflow.cfg
    User=ewsmanager
    Group=ewsmanager
    Type=simple
    ExecStart=/usr/bin/bash -c 'export AIRFLOW_HOME=/storage/airflow ; source  /storage/airflow/airflow-env/bin/activate ; airflow scheduler'
    Restart=no
    RestartSec=5s
    PrivateTmp=true
    [Install]
    WantedBy=multi-user.target

2: reload the service daemon, enable and start the services

sudo systemctl daemon-reload

sudo systemctl enable airflow-scheduler.service
sudo systemctl enable airflow-webserver.service

sudo systemctl start airflow-scheduler.service
sudo systemctl start airflow-webserver.service

sudo systemctl restart airflow-scheduler.service
sudo systemctl restart airflow-webserver.service

sudo systemctl stop airflow-scheduler.service
sudo systemctl stop airflow-webserver.service

3: check the status of the services

sudo systemctl status airflow-scheduler.service
sudo systemctl status airflow-webserver.service

4: check the logs

sudo journalctl -r -u airflow-scheduler.service
sudo journalctl -r -u airflow-webserver.service