diff --git a/docs/workflow/pypi.md b/docs/workflow/pypi.md new file mode 100644 index 0000000000000000000000000000000000000000..b8168cf1dfcf46bb321f1dd50b4990f0e3d3475e --- /dev/null +++ b/docs/workflow/pypi.md @@ -0,0 +1,233 @@ +# Publishing Python packages + +We publish some of our Python packages on [pypi.org](https://pypi.org/). This is +partially to be a good Python citizen but also because it means that we can make +use of `pip` to install them easily where we use them in other projects. + +A [dedicated +template](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/pypi-release.yml) +is available in our CI templates project which can be used to automate +publication of packages to PyPI. + +## A tale of two PyPIs + +There are two publicly available instances of PyPI which we will use in this +guide. One is the "main" PyPI found at [https://pypi.org/](https://pypi.org/) +and the second is the "test" instance at +[https://test.pypi.org/](https://test.pypi.org/). We have jobs which can publish +to either. Publishing to the "test" instance is useful if you're testing +packaging, etc. and don't want to actually make a "true" release. + +## Signing in to PyPI + +In 1Password we have account credentials for two "uis-devops-bot" accounts, one +for each PyPI. Each are set up with 2FA and the 2FA tokens are present in +1Password. They are both configured with +[devops-account-recovery@uis.cam.ac.uk](mailto:devops-account-recovery@uis.cam.ac.uk) +as the recovery email address. + +This account is the primary owner of all of our published Python packages. + +## Terminology + +The steps in this guide assume that the configuration described below is present +on the "master" branch or, at least, on some other protected branch. When we use +the term "release branch" we mean the branch which has the PyPI-related GitLab +CI configuration. For already published packages this is likely to be "master". +For packages which have not yet been published this may be the branch that +contains the to-be-reviewed configuration. + +## Publishing a brand new package + +There is a small amount of setup required to publish brand new Python package to +PyPI. + +### Make sure `setup.py` is suitable + +Ensure that the package has an appropriate `setup.py` which allows it to be +installed via `pip`. An example is below: + +```py +# setup.py contains configuration allowing this package to be installed into +# the Python environment. + +import os +from setuptools import setup, find_packages + +# Customisable fields placed here for easy editing. PyPI does not allow multiple +# uploads with the same name so the version number *must* be incremented on each +# publish. Try to use the semver convention when deciding which part of the +# version to increment. +PACKAGE_NAME="frobnicate" +PACKAGE_DESCRIPTION="Frobnicate wuggs and kikis" +PACKAGE_VERSION="0.0.1" +PACKAGE_URL="https://gitlab.developers.cam.ac.uk/uis/devops/lib/frobnicate/" + +def load_requirements(): + """ + Load requirements file and return non-empty, non-comment lines with leading + and trailing whitespace stripped. + + """ + with open(os.path.join(os.path.dirname(__file__), 'requirements.txt')) as f: + return [ + line.strip() for line in f + if line.strip() != '' and not line.strip().startswith('#') + ] + + +with open("README.md") as fobj: + long_description = fobj.read() + + +setup( + name=PACKAGE_NAME, + version=PACKAGE_VERSION, + author="University of Cambridge Information Services", + author_email=f"devops+{PACKAGE_NAME}@uis.cam.ac.uk", + description=PACKAGE_DESCRIPTION, + long_description=long_description, + long_description_content_type="text/markdown", + url=PACKAGE_URL, + packages=find_packages(), + install_requires=load_requirements(), + classifiers=[ + "Programming Language :: Python :: 3", + + # Note: this is our standard license for open projects. + "License :: OSI Approved :: MIT License", + ], +) +``` + +This should be suitable for most of our Python packages. It assumes there is a +`README.md` file which documents the product and a `requirements.txt` file which +specifies the dependencies. + +### Add PyPI jobs to GitLab CI + +We make use of GitLab CI to publish packages. We have a standard template +available which may be included into the project's `.gitlab-ci.yml` file as +follows: + +```yaml +# .gitlab-ci.yml contains configuration for GitLab CI jobs. + +include: + # ... other includes + + # Support uploading to PyPI + - project: 'uis/devops/continuous-delivery/ci-templates' + file: '/pypi-release.yml' +``` + +### Create a temporary test PyPI API token + +The PyPI publication jobs won't be added to pipelines unless the appropriate +`PYPY_API_TOKEN` or `TEST_PYPI_API_TOKEN` variables are defined in the GitLab CI +configuration and so the next step is to create an API token. + +PyPI allows API tokens to have "scopes" limiting them to certain projects. This +presents us with a bit of a chicken-and-egg problem: we want to use an API token +limited in scope to the project we're releasing but we can't *select* that scope +until we've published the project. We'll have to do a bit of token gymnastics to +get around this. + +Sign into the [test PyPI instance](https://test.pypi.org/) using the +`uis-devops-bot` account credentials. [Create a new API +token](https://test.pypi.org/manage/account/token/) with the following settings: + +* Name: **Temporary all scopes token to publish [PACKAGE NAME]** +* Scope: **Entire account (all projects)** + +Make sure to copy the API token somewhere safe since it will only be shown to +you once. + +### Add the temporary test PyPI API token as a GitLab CI variable + +In the CI/CD settings for the target project on GitLab, add a new variable: + +* Key: **TEST_PYPI_API_TOKEN** +* Value: _paste in API token_ +* Protect variable: **yes** +* Mask variable: **yes** + +!!!important + We have configured the variable as being *protected*. This means that it + will only be available to CI jobs running on [protected + branches](https://docs.gitlab.com/ee/user/project/protected_branches.html). + By default only "master" is a protected branch in our standard project + configuration. + +### Make the first release + +Trigger a CI/CD pipeline run for the release branch. (This can be done via the +**Run Pipeline** button on the CI/CD pipelines page for the GitLab project.) A +new [manual job](https://docs.gitlab.com/ee/ci/yaml/#whenmanual) is created +called `test-pypi-release`. + +Visit the pipeline page for the triggered pipeline and click the "play" button +next to `test-pypi-release`. Confirm that an initial release has been made to +the Test PyPI by looking in the log for the job. + +!!!info + If there is no `test-pypi-release` job, make sure that the branch the + pipeline is running on is protected. + +### Replace the API token with a scoped token + +On the [account management page](https://test.pypi.org/manage/account/) for test +PyPI, remove the temporary token created above and [create a new +one](https://test.pypi.org/manage/account/token/): + +* Name: **GitLab CI for [PACKAGE NAME]** +* Scope: *Select the appropriate project* + +Copy the new token and replace the value for the `TEST_PYPI_API_TOKEN` in the +GitLab CI/CD settings. + +### Test the new token + +Trigger a pipeline for the release branch and start the `test-pypi-release` job +as before. The publication will fail because PyPI does not allow uploading new +packages with the same version number but you shouldn't get a failure due to bad +authentication. + +### Repeat for the production PyPI + +Once you are happy publication is working well for the test PyPI, repeat the +instructions for the [production PyPI instance](https://pypi.org/). The steps +are the same except that tokens should be created in the main PyPI instance, +the GitLab CI/CD variable should be `PYPY_API_TOKEN` and the job is called +`pypi-release`. + +## Publishing a previously published package + +This is far simpler because some kind soul will already have performed the +configuration above. + + +### Bump the version number if necessary + +Before you publish make sure that the version of the package as set in +`setup.py` is greater than the version currently published on PyPI. We use +[semver](https://semver.org/) when versioning our packages. + +!!!tip + Python packages support adding "devN" to a version to indicate a development + release which should not be installed unless explicitly asked for. As such, + if the most recent published version is "1.2.3", you can publish "1.2.3dev4" + without it becoming the automatically installed version. + +### Publish to the test PyPI + +On the GitLab project page, click on the "pipeline status" badge to get to the +most recently run pipeline on "master". There should be a manual job waiting for +you there called `test-pypi-release`. Click the "play" button and check the job +logs to see if the package was successfully published. + +### Publish to production PyPI + +If the test publication succeeds, click the "play" button next to the +`pypi-release` job in the same pipeline. This should publish the package to the +production PyPI site. diff --git a/mkdocs.yml b/mkdocs.yml index 670ab0bcdff7483d1b8d7ad274e55a0f99060ad3..4b74d9c44d1b8721fa5b4cd5a5403c1b388ff5b3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -20,6 +20,7 @@ nav: - workflow/flows.md - workflow/merge-requests.md - workflow/onboarding.md + - workflow/pypi.md - 'Best Practice': - best-practice/index.md - best-practice/git.md