Logan

This repository contains the logan tool, which wraps the docker run command, bootstrapping your environment so that appropriate secrets are available. It also makes sure that the image is up-to-date with the checked-out deployment.

Packaged with logan is a separate tool named sanctuary which can be used to copy secrets from 1password to Google Cloud as part of a bootstrap process. This is documented in a separate document.

Requirements

• Python >= 3.8 to be installed.
• docker to be installed and working.
• Google application default credentials configured via, e.g., gcloud auth application-default login or the GOOGLE_APPLICATION_DEFAULT_CREDENTIALS environment variable.
• An appropriate SSH agent to be running with a key which can access gitlab.developers.cam.ac.uk. On OS X, when using Docker Desktop for Mac, add your key to the host's ssh agent with ssh-add

If you are developing logan, follow the instructions in the development section below for additional dependencies.

Installation

The tool can be installed via pip:

$ pip3 install logan --index-url https://gitlab.developers.cam.ac.uk/api/v4/projects/404/packages/pypi/simple

Resolving "Command Not Found" for Logan

If you encounter a “command not found” error after successfully running the above command and attempting to execute logan -v, follow these steps to resolve the issue:

1.	Use pipx for Installation: Install Logan using pipx to ensure the command is accessible in your environment. Execute the following command:

  $ pipx install logan --index-url https://gitlab.developers.cam.ac.uk/api/v4/groups/5/-/packages/pypi/simple

This should resolve the issue and allow you to use the logan command as expected.

Direct Installation

The tool can be also installed directly from the git repository:

$ pip3 install git+https://gitlab.developers.cam.ac.uk/uis/devops/tools/logan.git

For developers, the tool can be installed from a cloned repo using pip:

$ cd /path/to/this/repo
$ pip3 install -e .

Usage

For the use of the sanctuary tool, see the separate documentation page.

See following output of $ logan --help:

 usage: logan [-h] [-c CONFIGURATION] [--dry-run] [--quiet]
                   [--image-name IMAGE_NAME] [--nobuild] [--nopull]
                   [--workdir WORKDIR] [--writable-workdir] [--workspace WORKSPACE]
                   [--terraform-data-volume-name TERRAFORM_DATA_VOLUME_NAME]
                   [--rm-terraform-data-volume] [-T]
                   [--docker-run-args DOCKER_RUN_ARGS]
                   [--temp-directory TEMP_DIRECTORY]
                   [-fo FACTORY_ITEM]
                   [-fe FACTORY_ITEM]
                   ...

 Wrap docker run command

 positional arguments:
   cmdargs               Remainder of arguments are passed to command run by
                         docker run

 optional arguments:
   -h, --help            show this help message and exit
   -c CONFIGURATION, --configuration CONFIGURATION
                         Location of configuration file
   --dry-run             Do not actually decrypt secrets and run docker
   --quiet               Reduce logging verbosity
   --image-name IMAGE_NAME
                         Name of Docker image to use/build, or override image
                         to use/pull if "image" is specified in the configuration
                         file
   --nobuild             Disable (re-)building of container
   --nopull              Disable (re-)pulling of image
   --workdir WORKDIR     Override working directory inside the container (if an
                         image is used)
   --writable-workdir    Mount working directory as writable instead of read-only [1]
   --workspace WORKSPACE
                         Terraform workspace
   --terraform-data-volume-name TERRAFORM_DATA_VOLUME_NAME
                         Name of Docker volume to hold terraform data
                         directory.
   --rm-terraform-data-volume
                         Delete terraform data volume before running docker run
   -T, --notty           Do not attach a pseudo TTY to the input
   --docker-run-args DOCKER_RUN_ARGS
                         Additional arguments to docker run. Split using
                         shlex.split()
   --temp-directory TEMP_DIRECTORY
                         Override location temporary directory is created in
   -fo, --factory-only FACTORY_ITEM
                         Only run factory for item(s) eg --fo foo --fo bar
   -fe, --factory-exclude FACTORY_ITEM
                         Exclude item(s) from factory run

By default, workspace will be development.

[1] Making the working directory writable may result in root owned files being created and/or existing files being modified. Use with caution.

Some installations of Ubuntu have issues mounting files/directories from /tmp in to a docker container. Creating a ~/tmp and using the argument --temp-directory ~/tmp will workaround the issue.

Configuration file

Unless overridden on the command line by the configuration argument, the tool reads its configuration from a .logan.yaml file in the current directory.

See the example configuration file (.logan.example.yaml) for what this file should look like. It should contain a dictionary with:

• version - string specifying configuration version (default: "1.0")
• mount_docker_socket - boolean indicating if the Docker socket should be mounted inside the container (OS X and Linux only, configuration version >= 1.1, default: false)
• project - name of project/repo (mandatory)
• secrets - list of secret definitions (optional):
  • name - friendly name of the secret
  • source - source of secret. see below
  • target - where the decrypted secret should be mounted in the container
• image - docker image to pull (see below, optional)
• build - override options used for building image (optional)
  • dockerfile - custom Dockerfile location (optional, default is Dockerfile)
  • context - optional build context location
• env - dictionary of environment variables optionally with values (optional)
  • if no value is given for a variable then the matching shell variable will be passed through
• factory_mode - boolean, enable iterative factory mode for project, allows logan to launch a container with a seperate temp dir for each folder under factory_vars_path (to one level)
• factory_vars_path - The path to use for the factory loop, default ./product-vars
• factory_prefix - The prefix added to docker volumes defaults to project

Image Name

If no image value is specified in the .logan.yml configuration file then logan will build an image using the Dockerfile in the current directory. This image will be tagged with the project specified in the configuration file unless override by the --image-name command line argument. --nobuild will prevent a rebuild but continue to use the tagged image.

If image is specified in the configuration file then logan will pull this image (unless --nopull given) and use the image. --image-name can be used to override which image is pulled/used.

Specifying secrets

Logan can access secrets in the following manner:

• Google Secret Manager secrets.

Logan will use the current default credentials to decrypt the secret with the key. You need to make sure that the current user has permission to use the key.

To authenticate as your personal user, use gcloud auth application-default login.

Secret Managed secrets

For secret manager secrets, you need to specify the source, target and a friendly name.

The source should be formatted as a URL in the following form:

sm://<PROJECT>/<NAME>[#<VERSION>]

Where:

• PROJECT is the id of the Google Cloud project.
• NAME is the name of the secret.
• VERSION is the version of the secret. The default is latest if the URL fragment is not specified.

For example:

secrets:
  - name: some secret
    source: sm://my-project/super-secret
    target: /config/super-secret.txt

Logan will use the current default credentials to access the secret. You need to make sure that the current user has permission to access the secret.

To authenticate as your personal user, use gcloud auth application-default login.

Factory Mode

Logan will loop through the first level of the folder defined in factory_vars_path and launch a container for each with a seperate docker volume and temp directory for each item.

This allows for running the same code over several items whilst only swapping vars but keeping the logan environment seperate for each item

To enable set use_factory: true in the logan config for the project

The -fo option can be used to only run on the specified item, more than one can be specified by repeating eg -fo foo -fo bar

The -fe option can be used to exclude one item from the run, again multiple can be specified

Local development

Logan uses poetry for local building. To get started, make sure poetry is installed via pip install poetry.

Then install dependencies:

poetry install

Make sure pre-commit hooks are installed:

poetry run pre-commit install

Logan can be run via poetry:

poetry run logan --help

Tests can be run via tox:

poetry run tox

You can run all pre-commit tests via:

poetry run pre-commit run --all-files

You can bump the version number using the poetry version command. For example, to bump the patch version:

poetry version patch