- Mar 19, 2025
-
-
Dmitrii Unterov authored
-
Sebastiaan ten Pas authored
-
- Feb 28, 2025
-
-
Dmitrii Unterov authored
-
- Dec 03, 2024
-
-
Dr Rich Wareham authored
Bump the release-it version to support GitLab release assets for GitLab after 17.2.
-
- Nov 18, 2024
-
-
Dr Rich Wareham authored
We are looking to add Java pre-commit hooks to some repos[1] and, unsurprisingly, the Java-ecosystem linters are written in Java. The pre-commit image currently has no JRE installed in it but alpine package one and so adding one is easy enough. While we're here, bump the Python image version used for pre-commit v4 and provide a default for the `PYTHON_VERSION` build arg to squash a Docker build warning. Keep pre-commit v3 as is because that image should only be being used by older, possibly sparsely maintained, CI jobs and we don't want to switch the Python version out from under them. [1] https://gitlab.developers.cam.ac.uk/uis/devops/hr/hr-general/-/issues/29
-
- Nov 01, 2024
-
-
Dmitrii Unterov authored
-
- Oct 17, 2024
-
-
Dmitrii Unterov authored
-
Dr Rich Wareham authored
When pre-commit 4 was added, the "latest" tag wasn't removed from pre-commit-3.3 meaning that which ended up as "latest" depended on which would be the most recent image built by CI.
-
- Oct 16, 2024
-
-
Dr Rich Wareham authored
-
- Oct 14, 2024
-
-
Dr Rich Wareham authored
-
- Oct 09, 2024
-
-
UIS DevOps Renovate Bot authored
-
- Sep 10, 2024
-
-
Dmitrii Unterov authored
-
- Sep 02, 2024
-
-
Dmitrii Unterov authored
-
- Aug 23, 2024
-
-
Adam Deacon authored
-
Adam Deacon authored
-
- Jul 01, 2024
-
-
Roy Harrington authored
fix: gcloud-docker-435.0.1 was failing due to being incompatible with Python 3.12. Need to pin this to use 3.11
-
- Apr 09, 2024
-
-
Dmitrii Unterov authored
-
Dmitrii Unterov authored
-
- Apr 08, 2024
-
-
Dmitrii Unterov authored
-
- Jan 15, 2024
-
-
Robin Goodall authored
-
Robin Goodall authored
-
- Dec 19, 2023
-
-
Robin Goodall authored
-
- Dec 18, 2023
-
-
Robin Goodall authored
-
- Dec 12, 2023
-
-
Daniel Ball authored
-
- Oct 30, 2023
-
-
Dr Rich Wareham authored
Re-work all uses of envsubst into equivalent build arg-based solutions and remove envsubst all together. This had the knock-on effect of a) moving additional Python requirements specification out of files and into the dynamic pipeline config and b) allowed us to unify the installed dependencies between the Debian and alpine-based Python images.
-
- Oct 26, 2023
-
-
Dr Rich Wareham authored
Move to generating image build jobs for Django, Logan/terraform and Python dynamically by querying the appropriate upstream APIs. This is mostly a genericisation of the previous logan terraform work. We extend our version parsing code to understand PEP440-style versions and code in some knowledge about which Django versions are compatible with which Python versions. Aside from a fair amount of Python code, the only _changes_ to image building are as follows: * The Python alpine image has been updated to allow passing Django versions dynamically rather than having a `requirements/django-{...}.txt` file for each Django release. * We've had to put in some horrible hackery to support _both_ envsubst and build args for Docker image building. * Building of django, Python and logan/terraform images are serialised by means of a `resource_group` since, in testing, the number of dynamic jobs could easily overwhelm the VM runners. Anticipated Python version support changes for Django 5.0 have been coded in so that we don't get taken by surprise on Django 5.0 release day.
-
- Oct 25, 2023
-
-
Dr Rich Wareham authored
In preparation for generating Django build jobs automatically, genericise some of the infrastructure and documentation around the dynamic pipeline jobs.
-
- Oct 19, 2023
-
-
Ryan Kowalewski authored
-
Ryan Kowalewski authored
-
Ryan Kowalewski authored
This allows us to build images for new versions of Terraform as soon as they are released.
-
- Oct 05, 2023
-
-
Ryan Kowalewski authored
-
- Sep 25, 2023
-
-
Adam Deacon authored
-
- Aug 21, 2023
-
-
Dr Rich Wareham authored
Fix the shellcheck warnings in build.sh and add pre-commit checks to ensure that we don't add more in future.
-
Dr Rich Wareham authored
-
Dr Rich Wareham authored
When building multi platform images, it is unfortunately quite easy to accidentally build, for example, an x86_64 image when you meant to build an aarch64 one; if the base image specified in "FROM" doesn't have an aarch64 image available, docker will happily proceed and build an x86_64 one. We addressed this issue by pulling the images we just pushed back and running `uname --machine` inside them to check that the platform was what we expected. Unfortunately that behaviour was racy. There is a small delay between finishing a push or an image replacing one with an existing tag and the new image being available for pulling by that tag. This is perfectly reasonable behaviour on the part of the upstream registry but poses a bit of a problem for us in that, occasionally, we'd be pulling back the incorrect image for testing. It also had the slightly annoying feature of requiring that `uname` was present in the built images and that `uname --machine` had uniform output. In `FROM scratch` images packaging Go applications, for example, this is not generally the case. The fix would appear to be straightforward: pull the image by SHA256 digest. Unfortunately there are a few issues with this: - The SHA256 digests for images differ depending on their platform. As such we'd need to somehow know which platforms were built and the SHA256 digests for each image and knowing which platforms are built is the problem we're trying to solve. - The digest written to the build metadata and via the `--iidfile` option to `docker buildx build` correspond either to one image or a manifest list. Neither of those digests are useful in general to pull all images. We have to actually query what platforms were actually built in a non-racy way. For single platform images, this is fairly straightforward in that the os and architecture are written by buildx to the metadata. For multi platform images, things are a bit more complex in that the build metadata records the name and the SHA256 of the *list* of images pushed to the registry. Fortunately we can use the name and the SHA256 digest to get this list back from the registry via `docker manifest`. This is non-racy because we can use the SHA256 digest to get back exactly what was pushed. The manifest list handily includes the platform of each image within it. As a final little annoyance, unless one actually *pushes* the images built by `buildx`, there is no way which I can see to get this information out so we need to skip this entire step if we're not pushing images. The exact plumbing for this is tiresome and verbose but it can be found in `build.sh`. https://gitlab.developers.cam.ac.uk/uis/devops/infra/dockerimages/-/pipelines/382292 is an example pipeline where multi- and single-platform images are built and the actual platforms are successfully recovered. I've also confirmed that executing `build.sh` locally skips the check as expected. Closes #74
-
- Aug 15, 2023
-
-
Dr Rich Wareham authored
Add configuration to build Python and Django base images for aarch64. We skip some images which timeout when building under aarch64 because grpcio takes _forever_ to build. Closes #73
-
- Aug 14, 2023
-
-
Ryan Kowalewski authored
-
Ryan Kowalewski authored
Testing this process showed that multi-platform builds can be quite flaky or extremely slow, depending on the complexities of each image's Dockerfile etc. With this in mind, this commit leaves the default image type as `linux/amd64` but allows us to override each build job as required using the `IMAGE_ARCHS` variable. Initially, all `logan-terraform` jobs have been set to build both amd64 and arm64 image types as this is the main requirement at this time. Given the differences between the `docker build` and `docker buildx build` command usage, the previous job functions have been refactored as a standalone `build.sh` file. The logic is identical to the previous functions, however, the method is slightly different. This is mostly due to the `docker buildx` command not having `tag` sub-command, instead, it uses `--tag` and `--push` options on a single `docker buildx build` command to both build and push images in one go.
-
- Aug 11, 2023
-
-
Ryan Kowalewski authored
-
- Jul 24, 2023
-
-
Ryan Kowalewski authored
-