DevOps docker images
This repository contains configuration to build the UIS DevOps docker images. The images are built nightly/weekly via GitLabCI and pushed to this repository's Registry.
Images
The following image types are built. See the README files in each image's directory for more details.
- ansible-playbook
- commitlint
- python-alpine
- python-slim
- django
- ucam-webauth-proxy
- ucam-shib-proxy
- ucam-shib3-proxy
- ucam-mellon-proxy
- logan-terraform
- gcloud-docker
- photo-api-base
- pre-commit
- prometheus-remote-writer
- release-it
- terminus
Configuration
The GitLabCI configuration defines the images to be built. The jobs are triggered by a weekly or nightly scheduled pipeline and images are pushed to the Registry.
Dynamic pipeline configuration
The .gitlab-ci.yml
pipeline file contains a dynamic-pipeline-config
job which runs dynamic
pipeline generation scripts. These scripts dynamically add build jobs to the pipeline
for multiple versions of Terraform.
We have chosen to implement some builds using a dynamic pipeline for some images for the following reasons:
- We can continue to use the same base
.build_and_push
job from thebuild-and-push.gitlab-ci.yml
template. This ensures that all our images are built consistently, using the same process. - Build jobs can run in parallel. This ensures that single jobs can fail without, necessarily, causing the other jobs to fail. It also ensures that the pipeline completes as quickly as possible.
- Build jobs automatically pick up new minor and patch releases.
For terraform images, we use the Hashicorp Releases
API to query all available Terraform
versions and then add jobs for each minor version, e.g. logan-terraform-1.6
,
logan-terraform-1.5
etc. This ensures that we always build new versions of the image for each new
minor release of Terraform. It also ensures that each minor version that we build includes the
latest patch release available. For example, assuming that the logan-terraform-1.6
image had
previously been built using the 1.6.0
release of Terraform, when Hashicorp release version 1.6.1
this script would rebuild our logan-terraform-1.6
image using the newer 1.6.1
version of
Terraform the next time the pipeline runs.
Job variables
Each job can be configured using the following variables.
Name | Default Value | Description |
---|---|---|
PUSH_DOCKER | True | Push the resulting images to the uisautomation public Docker Hub registry. |
PUSH_GITLAB | True | Push to the container to this repository's container registry. |
PUSH_TEST_IMAGES | False | Useful to push individual images to this repository's registry for testing. |
ENVSUBST_DOCKERFILE | False | Use the envsubst command to substitute environment variables into the Dockerfile at build time. |
IMAGE_ARCHS | linux/amd64 | Specify the target platforms for the image. This is a comma-separated string, for example linux/amd64,linux/arm64 . |
Multi-architecture builds
Images are built using the docker buildx build
command which enables building
images for multiple architectures. However, during testing it was noted that
building linux/arm64
images on x86 runners can be flaky or simply take a very
long time. With this in mind, care should be taken when enabling
multi-architecture builds for a particular image type as Dockerfiles may need to
be refactored to overcome installation issues etc. which otherwise work fine
when building for linux/amd64
.
Creating test images
The GitLabCI configuration only pushes images to the local
repository if the branch is master
. To create images while testing, manually
start a new
pipeline,
select the branch to test and set PUSH_TEST_IMAGES
to True
. Images will then
be pushed to a test
subfolder of the
registry.
You can force dynamic CI pipeline jobs to run by setting FORCE_DYNAMIC_JOBS
when running the
pipeline.
Adding known vulnerabilities to allowlist file
This project has container-scanning job enabled for all images. As a result, this project has detailed
Vulnerability report
with information about all detected vulnerabilities. Periodically, the Cloud Team triage those findings and some of
them could be count as false positive or known issues. To exclude such issues from the final report, the issues must be
added to the vulnerability-allowlist.yml
file in the root of this project. The syntax is described in
GitLab docs.
generalallowlist:
# https://gitlab.developers.cam.ac.uk/uis/devops/infra/dockerimages/-/issues/80
CVE-2023-6879: libaom
images:
registry.gitlab.developers.cam.ac.uk/uis/devops/infra/dockerimages/ucam-mellon-proxy:
# https://gitlab.developers.cam.ac.uk/uis/devops/infra/dockerimages/-/issues/81
CVE-2023-45853: zlib1g
On the example above:
- CVE-2023-6879: allowed for every image where it is detected in
libaom
package; - CVE-2023-6879: allowed for image
registry.gitlab.developers.cam.ac.uk/uis/devops/infra/dockerimages/ucam-mellon-proxy
,zlib1g
package;
The GitLab issue reference must be added as a comment.
This file is also being used for "container-scanning" job in
CI Templates project.
The job downloads latest version of this file and use it as a source of information about allowed
vulnerabilities. Project's own vulnerability-allowlist.yml
file has priority over the one in this repository.