new page: publishing to PyPI

Add a new page describing how we publish to PyPI and how to set up new
projects to be published.

docs/workflow/pypi.md

+# Publishing Python packages
+
+We publish some of our Python packages on [pypi.org](https://pypi.org/). This is
+partially to be a good Python citizen but also because it means that we can make
+use of `pip` to install them easily where we use them in other projects.
+
+A [dedicated
+template](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/pypi-release.yml)
+is available in our CI templates project which can be used to automate
+publication of packages to PyPI.
+
+## A tale of two PyPIs
+
+There are two publicly available instances of PyPI which we will use in this
+guide. One is the "main" PyPI found at [https://pypi.org/](https://pypi.org/)
+and the second is the "test" instance at
+[https://test.pypi.org/](https://test.pypi.org/). We have jobs which can publish
+to either. Publishing to the "test" instance is useful if you're testing
+packaging, etc. and don't want to actually make a "true" release.
+
+## Signing in to PyPI
+
+In 1Password we have account credentials for two "uis-devops-bot" accounts, one
+for each PyPI. Each are set up with 2FA and the 2FA tokens are present in
+1Password. They are both configured with
+[devops-account-recovery@uis.cam.ac.uk](mailto:devops-account-recovery@uis.cam.ac.uk)
+as the recovery email address.
+
+This account is the primary owner of all of our published Python packages.
+
+## Terminology
+
+The steps in this guide assume that the configuration described below is present
+on the "master" branch or, at least, on some other protected branch. When we use
+the term "release branch" we mean the branch which has the PyPI-related GitLab
+CI configuration. For already published packages this is likely to be "master".
+For packages which have not yet been published this may be the branch that
+contains the to-be-reviewed configuration.
+
+## Publishing a brand new package
+
+There is a small amount of setup required to publish brand new Python package to
+PyPI.
+
+### Make sure `setup.py` is suitable
+
+Ensure that the package has an appropriate `setup.py` which allows it to be
+installed via `pip`. An example is below:
+
+```py
+# setup.py contains configuration allowing this package to be installed into
+# the Python environment.
+
+import os
+from setuptools import setup, find_packages
+
+# Customisable fields placed here for easy editing. PyPI does not allow multiple
+# uploads with the same name so the version number *must* be incremented on each
+# publish. Try to use the semver convention when deciding which part of the
+# version to increment.
+PACKAGE_NAME="frobnicate"
+PACKAGE_DESCRIPTION="Frobnicate wuggs and kikis"
+PACKAGE_VERSION="0.0.1"
+PACKAGE_URL="https://gitlab.developers.cam.ac.uk/uis/devops/lib/frobnicate/"
+
+def load_requirements():
+    """
+    Load requirements file and return non-empty, non-comment lines with leading
+    and trailing whitespace stripped.
+
+    """
+    with open(os.path.join(os.path.dirname(__file__), 'requirements.txt')) as f:
+        return [
+            line.strip() for line in f
+            if line.strip() != '' and not line.strip().startswith('#')
+        ]
+
+
+with open("README.md") as fobj:
+    long_description = fobj.read()
+
+
+setup(
+    name=PACKAGE_NAME,
+    version=PACKAGE_VERSION,
+    author="University of Cambridge Information Services",
+    author_email=f"devops+{PACKAGE_NAME}@uis.cam.ac.uk",
+    description=PACKAGE_DESCRIPTION,
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url=PACKAGE_URL,
+    packages=find_packages(),
+    install_requires=load_requirements(),
+    classifiers=[
+        "Programming Language :: Python :: 3",
+
+        # Note: this is our standard license for open projects.
+        "License :: OSI Approved :: MIT License",
+    ],
+)
+```
+
+This should be suitable for most of our Python packages. It assumes there is a
+`README.md` file which documents the product and a `requirements.txt` file which
+specifies the dependencies.
+
+### Add PyPI jobs to GitLab CI
+
+We make use of GitLab CI to publish packages. We have a standard template
+available which may be included into the project's `.gitlab-ci.yml` file as
+follows:
+
+```yaml
+# .gitlab-ci.yml contains configuration for GitLab CI jobs.
+
+include:
+  # ... other includes
+
+  # Support uploading to PyPI
+  - project: 'uis/devops/continuous-delivery/ci-templates'
+    file: '/pypi-release.yml'
+```
+
+### Create a temporary test PyPI API token
+
+The PyPI publication jobs won't be added to pipelines unless the appropriate
+`PYPY_API_TOKEN` or `TEST_PYPI_API_TOKEN` variables are defined in the GitLab CI
+configuration and so the next step is to create an API token.
+
+PyPI allows API tokens to have "scopes" limiting them to certain projects. This
+presents us with a bit of a chicken-and-egg problem: we want to use an API token
+limited in scope to the project we're releasing but we can't *select* that scope
+until we've published the project. We'll have to do a bit of token gymnastics to
+get around this.
+
+Sign into the [test PyPI instance](https://test.pypi.org/) using the
+`uis-devops-bot` account credentials. [Create a new API
+token](https://test.pypi.org/manage/account/token/) with the following settings:
+
+* Name: **Temporary all scopes token to publish [PACKAGE NAME]**
+* Scope: **Entire account (all projects)**
+
+Make sure to copy the API token somewhere safe since it will only be shown to
+you once.
+
+### Add the temporary test PyPI API token as a GitLab CI variable
+
+In the CI/CD settings for the target project on GitLab, add a new variable:
+
+* Key: **TEST_PYPI_API_TOKEN**
+* Value: _paste in API token_
+* Protect variable: **yes**
+* Mask variable: **yes**
+
+!!!important
+    We have configured the variable as being *protected*. This means that it
+    will only be available to CI jobs running on [protected
+    branches](https://docs.gitlab.com/ee/user/project/protected_branches.html).
+    By default only "master" is a protected branch in our standard project
+    configuration.
+
+### Make the first release
+
+Trigger a CI/CD pipeline run for the release branch. (This can be done via the
+**Run Pipeline** button on the CI/CD pipelines page for the GitLab project.) A
+new [manual job](https://docs.gitlab.com/ee/ci/yaml/#whenmanual) is created
+called `test-pypi-release`.
+
+Visit the pipeline page for the triggered pipeline and click the "play" button
+next to `test-pypi-release`. Confirm that an initial release has been made to
+the Test PyPI by looking in the log for the job.
+
+!!!info
+    If there is no `test-pypi-release` job, make sure that the branch the
+    pipeline is running on is protected.
+
+### Replace the API token with a scoped token
+
+On the [account management page](https://test.pypi.org/manage/account/) for test
+PyPI, remove the temporary token created above and [create a new
+one](https://test.pypi.org/manage/account/token/):
+
+* Name: **GitLab CI for [PACKAGE NAME]**
+* Scope: *Select the appropriate project*
+
+Copy the new token and replace the value for the `TEST_PYPI_API_TOKEN` in the
+GitLab CI/CD settings.
+
+### Test the new token
+
+Trigger a pipeline for the release branch and start the `test-pypi-release` job
+as before. The publication will fail because PyPI does not allow uploading new
+packages with the same version number but you shouldn't get a failure due to bad
+authentication.
+
+### Repeat for the production PyPI
+
+Once you are happy publication is working well for the test PyPI, repeat the
+instructions for the [production PyPI instance](https://pypi.org/). The steps
+are the same except that tokens should be created in the main PyPI instance,
+the GitLab CI/CD variable should be `PYPY_API_TOKEN` and the job is called
+`pypi-release`.
+
+## Publishing a previously published package
+
+This is far simpler because some kind soul will already have performed the
+configuration above.
+
+
+### Bump the version number if necessary
+
+Before you publish make sure that the version of the package as set in
+`setup.py` is greater than the version currently published on PyPI. We use
+[semver](https://semver.org/) when versioning our packages.
+
+!!!tip
+    Python packages support adding "devN" to a version to indicate a development
+    release which should not be installed unless explicitly asked for. As such,
+    if the most recent published version is "1.2.3", you can publish "1.2.3dev4"
+    without it becoming the automatically installed version.
+
+### Publish to the test PyPI
+
+On the GitLab project page, click on the "pipeline status" badge to get to the
+most recently run pipeline on "master". There should be a manual job waiting for
+you there called `test-pypi-release`. Click the "play" button and check the job
+logs to see if the package was successfully published.
+
+### Publish to production PyPI
+
+If the test publication succeeds, click the "play" button next to the
+`pypi-release` job in the same pipeline. This should publish the package to the
+production PyPI site.

mkdocs.yml

@@ -20,6 +20,7 @@ nav:
     - workflow/flows.md
     - workflow/merge-requests.md
     - workflow/onboarding.md
+    - workflow/pypi.md
   - 'Best Practice':
     - best-practice/index.md
     - best-practice/git.md