add python-package template
Add a template intended for standalone Python packages. The template supports building tarballs and wheels via either setuptools or poetry, running parallel tests via tox for multiple Python versions and uploading packages to the GitLab package registry and/or the PyPI.
See auto-devops/python.md for fuller documentation.
This also introduces a "common" pipeline template for Auto DevOps which brings in all of the "safe" templates. "Safe" in this instance means:
- They are only enabled if files in the repository indicate they are required or if appropriate variables are set.
- They can be disabled by setting a variable if they are inappropriate for a given project.
The Python and pre-commit templates are "safe" in that definition and so they are added. There was a degree of duplication of the "rules:" portions of those jobs and so they have been broken out into a common include file.
Adding this to existing projects
Generally this involves adding the following trivial .gitlab-ci.yml file:
include:
- project: 'uis/devops/continuous-delivery/ci-templates'
file: '/auto-devops/common-pipeline.yml'Logan
Using the logan tool as an example:
An example of a push pipeline when there is an open MR is at https://gitlab.developers.cam.ac.uk/uis/devops/tools/logan/-/pipelines/350198.
An example of a merge request pipeline is at https://gitlab.developers.cam.ac.uk/uis/devops/tools/logan/-/pipelines/350199.
An example of a push pipeline when there was no corresponding MR is at https://gitlab.developers.cam.ac.uk/uis/devops/tools/logan/-/pipelines/350200. Note that this pipeline runs the packaging and tox tests.
An example of a tag pipeline when the tag matches the package version is at https://gitlab.developers.cam.ac.uk/uis/devops/tools/logan/-/pipelines/350201.
An example of a tag pipeline when the tag does not match the package version is at https://gitlab.developers.cam.ac.uk/uis/devops/tools/logan/-/pipelines/350206. Note that this pipeline is intended to fail and thereby not attempt to publish the package.
Django "externalsettings" library
Using the external settings Django library. The required changes are at uis/devops/django/externalsettings@a74170b3.
Note that external settings is a setuptools-based project, not Poetry. The jobs detect the difference.
An example of a push pipeline running tests in multiple Python versions and multiple Django versions: https://gitlab.developers.cam.ac.uk/uis/devops/django/externalsettings/-/pipelines/350231
An example of a push pipeline for a new tag showing a new release happening via setuptools: https://gitlab.developers.cam.ac.uk/uis/devops/django/externalsettings/-/pipelines/350236
FastAPI library
Using the FastAPI pagination library, I opted to do the minimum integration at uis/devops/lib/fastapi-pagination@259d8f91. Note that this does not bring in pre-commit; it is a small change to support the new pipeline.
An example of a push pipeline running the tests in the three most recent Python versions: https://gitlab.developers.cam.ac.uk/uis/devops/lib/fastapi-pagination/-/pipelines/350242
Our webapp boilerplate
Again, a very minimal integration at https://gitlab.developers.cam.ac.uk/digital/site-management-app/webapp/-/commit/2afe0924fe2e8eafe21b653fc2757caa666f4d05 using the Site Management webapp as an example. I've not bothered using pre-commit, putting code coverage files in the right place or splitting the toxenvs up but those things are trivial busywork to do in a real change over.
This is more interesting since we need a Postgres database and some bespoke configuration to run the tests. Rather than having that sit entirely in the CI template, move that to the tox configuration using the tox-docker utility. This has the nice advantage that one can now just run tox in the terminal locally rather than having to use the wrapper stuff in tox.sh.
An example of a push pipeline running the tests using the common pipeline: https://gitlab.developers.cam.ac.uk/digital/site-management-app/webapp/-/pipelines/350281