FAQ | This is a LIVE service | Changelog

Skip to content
Snippets Groups Projects
Select Git revision
  • main default protected
1 result
Blame Permalink
Code owners
Assign users and groups as approvers for specific file changes. Learn more.
iaas_setup.rst 21.29 KiB

IaaS setup

This isnt included in the main docs, only linkable with a direct URL

create new Ubuntu instance using the IaaS broker service:

https://v-vra01.srv.uis.private.cam.ac.uk/catalog/#

latest instance is:

ssh lb584@10.136.11.3 BOT-cags1

mount storage

mount our RDS and RFS storage

the mounting of RDS and RFS is a workaround until we can purchase block storage on the IaaS. it works, but does not preserve different user permissions and is slow.

sudo apt-get install cifs-utils sudo apt install smbclient

create dirs to mount: sudo mkdir /mnt/RDS sudo mkdir /mnt/RFS

add to /etc/fstab

//hpc-isi-w.hpc.private.cam.ac.uk/rfs-x2jwbL2P9T4 /mnt/RFS cifs credentials=/root/.rfs_credentials,workgroup=BLUE.CAM.AC.UK,file_mode=0775,dir_mode=0775,uid=1006,gid=1008 0 0 lb584@rds.uis.cam.ac.uk:rds-ews_production-rjVtFkyQ1T0/ /mnt/RDS/ fuse.sshfs defaults,noauto,_netdev,allow_other,uid=1006,gid=1008 0 0

notes: uid and gid are for the ewsmanager user, which has not actually been made yet, and will have a different id on a new install

the sshfs mount will not get mounted automatically (has the “noauto” option) and will require this to mount:

sudo mount /mnt/RDS - and will prompt for the password. note that this is being mounted as lb584, a different owner will need to supply different credentials

the credentials stored for the RFS in /root/.rfs_credentials are also going to need to be adjusted to the appropriate user is lb584 is no longer about. It doesn't matter who mounts them (provided they have access to the storage set by the datamanager of the storage) as they both have the uid set to be the shared ewsmanager user.

run sudo dpkg-reconfigure tzdata to make sure it automatically adjusts the clock for BST/GMT Basic user and groups setup

1: install needed packages

initial login must be as root

apt-get update apt-get install postfix apt-get install emacs

open emacs and set the theme: alt-x customize-themes

2: create users

sudo adduser lb584,jws52,tm689,ewsmanager

3: add ews group

sudo groupadd ews

4: add users to the ews group

sudo usermod -aG ews ewsmanager/jwc52/tm689

5: allow members of ews group to su into ewsmanager user:

sudo emacs -nw /etc/pam.d/su

# This allows root to su without passwords (normal operation) auth sufficient pam_rootok.so

auth [success=ignore default=1] pam_succeed_if.so user = ewsmanager auth sufficient pam_succeed_if.so use_uid user ingroup ews

6: grant root users sudo privileges

usermod -aG sudo lb584

7: Install docker

download .deb from https://docs.docker.com/desktop/install/ubuntu/

or

wget -O docker-desktop-4.14.1-amd64.deb2 https://desktop.docker.com/linux/main/amd64/docker-desktop-4.14.1-amd64.deb?utm_source=docker&utm_medium=webreferral&utm_campaign=docs-driven-download-linux-amd64

install docker engine: (instructions at https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository)

sudo apt-get install ca-certificates curl gnupg lsb-release sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

add docker users: sudo groupadd docker sudo usermod -aG docker $USER (reconnect/reopen terminal)

check docker works:

<sudo> docker run hello-world (wont need sudo if the user is added to the docker group)

Add sftp access for metoffice

groupadd sftponly ${EDITOR:-nano} /etc/ssh/sshd_config

Match Group sftponly PermitTunnel no AllowAgentForwarding no AllowTcpForwarding no X11Forwarding no ChrootDirectory /storage/sftp/%u ForceCommand internal-sftp

service ssh start mkdir -p /storage/sftp/ chown -R root:root /storage/sftp/ chmod -R 755 /storage/sftp/

###REPEAT THIS PROCESS FOR THE PLANTVILAGE USER#### adduser --no-create-home --shell /usr/sbin/nologin metofficeupload # create the user and an associated group with the same name adduser metofficeupload sftponly # add to existing sftponly group to allow (only) the sftp subsystem to be used under the chroot mkdir -p /storage/sftp/metofficeupload/upload

chown root:root /storage/sftp/metofficeupload chown metofficeupload:metofficeupload /storage/sftp/metofficeupload/upload chmod -R 755 /storage/sftp/metofficeupload mkdir -p /storage/sftp/metofficeupload/upload/Ethiopia/fromMO/daily_name mkdir -p /storage/sftp/metofficeupload/upload/Ethiopia/toMO/

sftp metofficeupload@10.136.11.3

add the ewsmanager user to metofficeupload group so it can manage files written by this user

sudo usermod -aG metofficeupload ewsmanager ###END REPEAT THIS PROCESS FOR THE PLANTVILAGE USER####

Deploy and test the EWS code

1: add the ssh key of the server to gitlab for access.

generate ssh key (if dont have one already) ssh-keygen -t rsa -b 4096 -C <key_id> https://gitlab.developers.cam.ac.uk/-/profile/keys

2: make dirs for the EWS app

mkdir /storage/app/EWS_prod sudo chown -R ewsmanager:ews EWS_prod/ sudo chmod -R g+s EWS_prod/ sudo chmod -R g+w EWS_prod/

3: follow the deployment instructions in this doc.

https://docs.google.com/document/d/1nW0eZJoLLOFzb3Yp4OonhwrbOS2UFELok_QQflb-EUc/edit?usp=sharing Install and setup the apache server (file downloads and ews_browser)

setup dirs for apache:

sudo mkdir /storage/webdata/Ethiopia sudo mkdir /storage/webdata/SouthAsia sudo chown ewsmanager:ews SouthAsia/ Ethiopia/ sudo chmod g+ws Ethiopia/ sudo chmod g+ws SouthAsia/

sudo mkdir /storage/app/ews_browser sudo chown ewsmanager:ews ews_browser sudo chmod g+ws ews_browser

ln -s /storage/webdata/Ethiopia /var/www/html/Ethiopia ln -s /storage/webdata/SouthAsia /var/www/html/SouthAsia

install apache and other libs:

apt-get -qq install --assume-yes apache2 apache2-dev apache2-utils ssl-cert libapache2-mod-wsgi openssh-server

(if you get prompted about a newer version of the sshd_conf file being available, keep the current one as we modified this already.)

install miniconda:

export CONDA_DIR='/home/miniconda3'

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /home/miniconda.sh /bin/bash /home/miniconda.sh -b -p $CONDA_DIR

export PATH=$CONDA_DIR/bin:$PATH conda init bash

create a file called ews_browser.yml and paste the following:

name: /home/conda/ews_browser channels:

  • conda-forge
  • defaults
dependencies:
  • _libgcc_mutex=0.1=conda_forge
  • _openmp_mutex=4.5=2_gnu
  • ca-certificates=2022.6.15=ha878542_0
  • click=8.1.3=py38h578d9bd_0
  • flask=2.2.2=pyhd8ed1ab_0
  • importlib-metadata=4.11.4=py38h578d9bd_0
  • itsdangerous=2.1.2=pyhd8ed1ab_0
  • jinja2=3.1.2=pyhd8ed1ab_1
  • ld_impl_linux-64=2.36.1=hea4e1c9_2
  • libffi=3.4.2=h7f98852_5
  • libgcc-ng=12.1.0=h8d9b700_16
  • libgomp=12.1.0=h8d9b700_16
  • libnsl=2.0.0=h7f98852_0
  • libsqlite=3.39.2=h753d276_1
  • libstdcxx-ng=12.1.0=ha89aaad_16
  • libzlib=1.2.12=h166bdaf_2
  • markupsafe=2.1.1=py38h0a891b7_1
  • ncurses=6.3=h27087fc_1
  • openssl=3.0.5=h166bdaf_1
  • pip=22.2.2=pyhd8ed1ab_0
  • python=3.8.12=h0744224_3_cpython
  • python_abi=3.8=2_cp38
  • readline=8.1.2=h0f457ee_0
  • setuptools=65.3.0=pyhd8ed1ab_1
  • sqlite=3.39.2=h4ff8645_1
  • tk=8.6.12=h27826a3_0
  • werkzeug=2.2.2=pyhd8ed1ab_0
  • wheel=0.37.1=pyhd8ed1ab_0
  • xz=5.2.6=h166bdaf_0
  • zipp=3.8.1=pyhd8ed1ab_0
  • pip: - mod-wsgi==4.9.3

prefix: /home/conda/ews_browser

conda env create -f /home/ews_browser.yml -p /home/conda_envs/ews_browser

(faff) modify the wsgi.load script in /etc/apache2/mods-available to point to the same python as used by the ews_browser

Enable the conda env for the project:

conda activate /home/conda_envs/ews_browser

which mod_wsgi-express

gives you:

/home/conda_envs/ews_browser/bin/mod_wsgi-express

sudo /home/conda_envs/ews_browser/bin/mod_wsgi-express install-module

gives you:

LoadModule wsgi_module "/usr/lib/apache2/modules/mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so" WSGIPythonHome "/home/conda_envs/ews_browser"

put the above line as the text in:

/etc/apache2/mods-available/wsgi.load (replacing what is currently there)

This means that apache wsgi will use the version of python that is used by the app to which it is binding.

prepare certificates and passwords for apache:

copy the default-ssl.conf file into /etc/apache2/sites_available. See notes on installing genuine certificates below (once the server is up and running).

<IfModule mod_ssl.c>
<VirtualHost _default_:443>

ServerAdmin webmaster@localhost

DocumentRoot /var/www/html

ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined SSLEngine on

SSLCertificateFile /etc/apache2/ssl/server.crt SSLCertificateKeyFile /etc/apache2/ssl/server.key

<FilesMatch ".(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars

</FilesMatch> <Directory /usr/lib/cgi-bin>

SSLOptions +StdEnvVars

</Directory>

<Directory "/var/www/html/Ethiopia">
AuthType Basic AuthName "Restricted Content" AuthUserFile /etc/apache2/.htpasswd Require user ethiopia Options +Indexes +FollowSymLinks +Includes +MultiViews

</Directory> <Directory "/var/www/html/SouthAsia">

AuthType Basic AuthName "Restricted Content" AuthUserFile /etc/apache2/.htpasswd Require user southasia Options +Indexes +FollowSymLinks +Includes +MultiViews

</Directory>

TypesConfig /etc/mime.types

</VirtualHost>

</IfModule>

enable url rewriting in apache:

This allows a maintenance page to be displayed if services are down.

a2enmod rewrite;

edit the apache conf at /etc/apache2/apache2.conf

<Directory /var/www/>
Options Indexes FollowSymLinks AllowOverride All Require all granted

</Directory>

add a maintenance.html file to /var/www/html e.g.

<!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml">

<head>
<STYLE type="text/css">
body {font-family:sans-serif; color: black; background: white;} div {text-align: center}

</STYLE> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <title>Willow server homepage</title>

</head> <body>

<center>
<br><br> <div><i>This service is now running at <a href="https://epi.plantsci.cam.ac.uk/">https://epi.plantsci.cam.ac.uk</a> - please edit your url and bookmarks accordingly.</i></div>

</center>

</body>

</html>

add an .htaccess file in /var/www/html/.htaccess

ErrorDocument 503 /maintenance.html RewriteEngine On RewriteCond %{REMOTE_ADDR} !^000.000.000.000 RewriteCond %{REQUEST_URI} !/maintenance.html$ [NC] RewriteRule .* - [L,R=503]

The RewriteEngine is set to “Off” when not redirecting

modify the ews browser wsgi conf files to point to the html directory, rather than the code:

/etc/apache2/sites-available/ews_browser_client_africa.conf /etc/apache2/sites-available/ews_browser_client_asia.conf

comment in the directory mapping in place of the WSGI mapping. e.g.

WSGIDaemonProcess ews_browser_asia user=ewsmanager group=ewsmanager threads=5 python-home=/home/conda_envs/ews_browser/

WSGIScriptAlias /ews_browser_asia /storage/app/ews_browser/code/src/main/python/ews_browser_asia.wsgi <Directory "/storage/app/ews_browser/code/src/main/python/">

WSGIProcessGroup ews_browser_asia WSGIScriptReloading On WSGIApplicationGroup %{RESOURCE} Require all granted

</Directory> ErrorLog /storage/app/ews_browser/outputs/ews_browser_client_error.log

# IF THE SITE IS DOWN FOR MAINTENANCE, COMMENT THIS BLOCK IN AND THE ONE ABOVE OUT - IT WILL USE THE REDIRECT IN .HTACCESS TO DISPLAY THE MAINTENANCE MESSAGE #<Directory /var/www/> # Options Indexes FollowSymLinks # AllowOverride All # Require all granted #</Directory>

restart the server to implement the changes:

sudo service apache2 restart

This will make all traffic to the /var/www/html dir redirect to the maintenance page.

enable ssl in apache:

ln -s /etc/apache2/sites-available/default-ssl.conf /etc/apache2/sites-enabled/000-default-ssl.conf

a2enmod ssl (possible this is already enabled)

add passwords for Ethiopia and SouthAsia users:

touch /etc/apache2/.htpasswd

(get the password from someone or make a new one) htpasswd /etc/apache2/.htpasswd southasia htpasswd /etc/apache2/.htpasswd ethiopia

create server certificates - once you have a domain name registered and pointing at this instance:

sudo snap install --classic certbot (one off, to install software)

sudo ln -s /snap/bin/certbot /usr/bin/certbot

then generate the certificates using:

sudo certbot certonly --apache

(enter the domain name you want to register (which needs to be wired through to this instance)

note that this retrieves the certificate only, does not install them

This created cert files in:

/etc/letsencrypt/live/epi.plantsci.cam.ac.uk (where epi.plantsci.cam.ac.uk is the registered domain (in this example))

I then edited the certificate path lines in this file:

/etc/apache2/sites-enabled/ssl.conf

to:

SSLCertificateFile /etc/letsencrypt/live/epi.plantsci.cam.ac.uk/fullchain.pem

SSLCertificateKeyFile /etc/letsencrypt/live/epi.plantsci.cam.ac.uk/privkey.pem

These certificates should be auto-renewed as certbot sets up a timer task to check if renewal is needed. See the timer by running:

systemctl list-timers

make the output dirs for the pipeline:

mkdir -p /storage/app/ews_browser/outputs/temp_unzip_dir chmod -R 775 /storage/app/ews_browser/outputs/ chmod g+sw /storage/app/ews_browser/outputs/

deploy the ews_browser code

copy the ews_browser code into /storage/app/ews_browser

create a symlink to the wsgi conf files

ln -s /storage/app/ews_browser/code/src/main/python/ews_browser_client_asia.conf ews_browser_client_asia.conf ln -s /storage/app/ews_browser/code/src/main/python/ews_browser_client_africa.conf ews_browser_client_africa.conf

activate the sites:

a2ensite ews_browser_client_africa.conf a2ensite ews_browser_client_asia.conf

start server:

service apache2 start

setup gitlab runner for CI

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

note: on the IaaS I had to “mkdir /var/lib/gitlab-runner” as the logs were complaining and the service wasnt staying up. I'm hoping this was because I upgraded the runner version and something got out of whack.

sudo gitlab-runner register go into /etc/gitlab-runner as root and edit the newly registered runner in config.toml :

concurrent = 1 check_interval = 0

[[runners]]

name = "bot-cags1-production" url = "https://gitlab.developers.cam.ac.uk/" token = "sjgyL15rgdC1veHs6yPT" executor = "docker" [runners.docker]

tls_verify = false image = "python" privileged = false disable_cache = false volumes = ["/storage/webdata:/storage/moved","/storage/sftp:/storage/sftp","/storage/app/EWS_prod/envs/credentials:/storage/app/EWS_prod/envs/credentials","/storage/app/EWS_prod/regions:/storage/app/EWS_prod/regions","/storage/app/EWS_prod/code:/storage/app/EWS_prod/code","/cache"] shm_size = 0

[runners.cache]

Airflow installation

Install mysql:

https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-20-04

sudo apt install mysql-server

sudo systemctl start mysql.service

hack to allow root access via sudo on ubuntu (in the digital ocean docs)

sudo mysql ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password_here'; exit

sudo mysql_secure_installation (no opinion on what options are selected, except disabling root access from remote host (yes))

finish the hack for ubuntu root access: mysql -u root -p ALTER USER 'root'@'localhost' IDENTIFIED WITH auth_socket; exit </end ubuntu hack>

Install Airflow

https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

mkdir <path>/airflow cd airflow set the airflow home location export AIRFLOW_HOME=<path>/airflow

python3 -m venv airflow-env; source airflow-env/bin/activate;

pip install wheel; pip install apache-airflow[celery]==2.7.1 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.8.txt"

note that there are two dashes before constraint above, not a single long dash

pip check

install the docker operator

pip install apache-airflow-providers-docker

install mysql (need root privileges)

https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-20-04

sudo apt update; sudo apt install mysql-server; sudo systemctl start mysql.service;

follow the additional instructions to work around the mysql_secure_installation issue

sudo mysql; ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '<put_root_password_here>'; exit;

The password above has been set to the same as the ewsmanager password. You can now run the security manager script:

sudo mysql_secure_installation;

<choose sensible security options at your discretion>

Setup mysql database:

mysql -u root -p

CREATE DATABASE airflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;