FAQ | This is a LIVE service | Changelog

Skip to content
Snippets Groups Projects
  1. Mar 19, 2025
  2. Feb 28, 2025
  3. Dec 03, 2024
  4. Nov 18, 2024
    • Dr Rich Wareham's avatar
      feat: add a Java Runtime Environment to pre-commit image · fcf62cd4
      Dr Rich Wareham authored
      We are looking to add Java pre-commit hooks to some repos[1] and,
      unsurprisingly, the Java-ecosystem linters are written in Java. The
      pre-commit image currently has no JRE installed in it but alpine package
      one and so adding one is easy enough.
      
      While we're here, bump the Python image version used for pre-commit v4
      and provide a default for the `PYTHON_VERSION` build arg to squash a
      Docker build warning. Keep pre-commit v3 as is because that image should
      only be being used by older, possibly sparsely maintained, CI jobs and
      we don't want to switch the Python version out from under them.
      
      [1] https://gitlab.developers.cam.ac.uk/uis/devops/hr/hr-general/-/issues/29
      fcf62cd4
  5. Nov 01, 2024
  6. Oct 17, 2024
  7. Oct 16, 2024
  8. Oct 14, 2024
  9. Oct 09, 2024
  10. Sep 10, 2024
  11. Sep 02, 2024
  12. Aug 23, 2024
  13. Jul 01, 2024
  14. Apr 09, 2024
  15. Apr 08, 2024
  16. Jan 15, 2024
  17. Dec 19, 2023
  18. Dec 18, 2023
  19. Dec 12, 2023
  20. Oct 30, 2023
    • Dr Rich Wareham's avatar
      feat: remove use of envsubst · 60530e0a
      Dr Rich Wareham authored
      Re-work all uses of envsubst into equivalent build arg-based solutions
      and remove envsubst all together. This had the knock-on effect of a)
      moving additional Python requirements specification out of files and
      into the dynamic pipeline config and b) allowed us to unify the
      installed dependencies between the Debian and alpine-based Python
      images.
      60530e0a
  21. Oct 26, 2023
    • Dr Rich Wareham's avatar
      feat: dynamically generate Django, Python and logan/terraform jobs · 0b3745d7
      Dr Rich Wareham authored
      Move to generating image build jobs for Django, Logan/terraform and
      Python dynamically by querying the appropriate upstream APIs.
      
      This is mostly a genericisation of the previous logan terraform work. We
      extend our version parsing code to understand PEP440-style versions and
      code in some knowledge about which Django versions are compatible with
      which Python versions.
      
      Aside from a fair amount of Python code, the only _changes_ to image
      building are as follows:
      
      * The Python alpine image has been updated to allow passing Django
        versions dynamically rather than having a
        `requirements/django-{...}.txt` file for each Django release.
      * We've had to put in some horrible hackery to support _both_ envsubst
        and build args for Docker image building.
      * Building of django, Python and logan/terraform images are serialised
        by means of a `resource_group` since, in testing, the number of
        dynamic jobs could easily overwhelm the VM runners.
      
      Anticipated Python version support changes for Django 5.0 have been
      coded in so that we don't get taken by surprise on Django 5.0 release
      day.
      0b3745d7
  22. Oct 25, 2023
  23. Oct 19, 2023
  24. Oct 05, 2023
  25. Sep 25, 2023
  26. Aug 21, 2023
    • Dr Rich Wareham's avatar
      fix: fix shellcheck warnings in build.sh · de161c2d
      Dr Rich Wareham authored
      Fix the shellcheck warnings in build.sh and add pre-commit checks to
      ensure that we don't add more in future.
      de161c2d
    • Dr Rich Wareham's avatar
      wip · fa67f880
      Dr Rich Wareham authored
      fa67f880
    • Dr Rich Wareham's avatar
      fix: close race when checking image platforms · 72e3d986
      Dr Rich Wareham authored
      When building multi platform images, it is unfortunately quite easy to
      accidentally build, for example, an x86_64 image when you meant to build
      an aarch64 one; if the base image specified in "FROM" doesn't have an
      aarch64 image available, docker will happily proceed and build an x86_64
      one.
      
      We addressed this issue by pulling the images we just pushed back and
      running `uname --machine` inside them to check that the platform was
      what we expected.
      
      Unfortunately that behaviour was racy. There is a small delay between
      finishing a push or an image replacing one with an existing tag and the
      new image being available for pulling by that tag. This is perfectly
      reasonable behaviour on the part of the upstream registry but poses a
      bit of a problem for us in that, occasionally, we'd be pulling back the
      incorrect image for testing.
      
      It also had the slightly annoying feature of requiring that `uname` was
      present in the built images and that `uname --machine` had uniform
      output. In `FROM scratch` images packaging Go applications, for example,
      this is not generally the case.
      
      The fix would appear to be straightforward: pull the image by SHA256
      digest. Unfortunately there are a few issues with this:
      
      - The SHA256 digests for images differ depending on their platform. As
        such we'd need to somehow know which platforms were built and the
        SHA256 digests for each image and knowing which platforms are built is
        the problem we're trying to solve.
      - The digest written to the build metadata and via the `--iidfile`
        option to `docker buildx build` correspond either to one image or a
        manifest list. Neither of those digests are useful in general to pull
        all images.
      
      We have to actually query what platforms were actually built in a
      non-racy way.
      
      For single platform images, this is fairly straightforward in that the
      os and architecture are written by buildx to the metadata.
      
      For multi platform images, things are a bit more complex in that the
      build metadata records the name and the SHA256 of the *list* of images
      pushed to the registry. Fortunately we can use the name and the SHA256
      digest to get this list back from the registry via `docker manifest`.
      This is non-racy because we can use the SHA256 digest to get back
      exactly what was pushed. The manifest list handily includes the
      platform of each image within it.
      
      As a final little annoyance, unless one actually *pushes* the images
      built by `buildx`, there is no way which I can see to get this
      information out so we need to skip this entire step if we're not pushing
      images.
      
      The exact plumbing for this is tiresome and verbose but it can be found
      in `build.sh`.
      https://gitlab.developers.cam.ac.uk/uis/devops/infra/dockerimages/-/pipelines/382292
      is an example pipeline where multi- and single-platform images are built
      and the actual platforms are successfully recovered. I've also confirmed
      that executing `build.sh` locally skips the check as expected.
      
      Closes #74
      72e3d986
  27. Aug 15, 2023
  28. Aug 14, 2023
    • Ryan Kowalewski's avatar
    • Ryan Kowalewski's avatar
      feat: add multi platform build support · ae784e50
      Ryan Kowalewski authored
      Testing this process showed that multi-platform builds can be quite flaky or extremely
      slow, depending on the complexities of each image's Dockerfile etc. With this in mind,
      this commit leaves the default image type as `linux/amd64` but allows us to override
      each build job as required using the `IMAGE_ARCHS` variable.
      
      Initially, all `logan-terraform` jobs have been set to build both amd64 and arm64 image
      types as this is the main requirement at this time.
      
      Given the differences between the `docker build` and `docker buildx build` command usage,
      the previous job functions have been refactored as a standalone `build.sh` file. The logic
      is identical to the previous functions, however, the method is slightly different. This is
      mostly due to the `docker buildx` command not having `tag` sub-command, instead, it uses
      `--tag` and `--push` options on a single `docker buildx build` command to both build and
      push images in one go.
      ae784e50
  29. Aug 11, 2023
  30. Jul 24, 2023
Loading