FAQ | This is a LIVE service | Changelog

Commit 2a411c1c authored by Dr Rich Wareham's avatar Dr Rich Wareham
Browse files

new page: publishing to PyPI

Add a new page describing how we publish to PyPI and how to set up new
projects to be published.
parent 6b488744
Pipeline #37969 passed with stage
in 40 seconds
# Publishing Python packages
We publish some of our Python packages on [pypi.org](https://pypi.org/). This is
partially to be a good Python citizen but also because it means that we can make
use of `pip` to install them easily where we use them in other projects.
A [dedicated
is available in our CI templates project which can be used to automate
publication of packages to PyPI.
## A tale of two PyPIs
There are two publicly available instances of PyPI which we will use in this
guide. One is the "main" PyPI found at [https://pypi.org/](https://pypi.org/)
and the second is the "test" instance at
[https://test.pypi.org/](https://test.pypi.org/). We have jobs which can publish
to either. Publishing to the "test" instance is useful if you're testing
packaging, etc. and don't want to actually make a "true" release.
## Signing in to PyPI
In 1Password we have account credentials for two "uis-devops-bot" accounts, one
for each PyPI. Each are set up with 2FA and the 2FA tokens are present in
1Password. They are both configured with
as the recovery email address.
This account is the primary owner of all of our published Python packages.
## Terminology
The steps in this guide assume that the configuration described below is present
on the "master" branch or, at least, on some other protected branch. When we use
the term "release branch" we mean the branch which has the PyPI-related GitLab
CI configuration. For already published packages this is likely to be "master".
For packages which have not yet been published this may be the branch that
contains the to-be-reviewed configuration.
## Publishing a brand new package
There is a small amount of setup required to publish brand new Python package to
### Make sure `setup.py` is suitable
Ensure that the package has an appropriate `setup.py` which allows it to be
installed via `pip`. An example is below:
# setup.py contains configuration allowing this package to be installed into
# the Python environment.
import os
from setuptools import setup, find_packages
# Customisable fields placed here for easy editing. PyPI does not allow multiple
# uploads with the same name so the version number *must* be incremented on each
# publish. Try to use the semver convention when deciding which part of the
# version to increment.
PACKAGE_DESCRIPTION="Frobnicate wuggs and kikis"
def load_requirements():
Load requirements file and return non-empty, non-comment lines with leading
and trailing whitespace stripped.
with open(os.path.join(os.path.dirname(__file__), 'requirements.txt')) as f:
return [
line.strip() for line in f
if line.strip() != '' and not line.strip().startswith('#')
with open("README.md") as fobj:
long_description = fobj.read()
author="University of Cambridge Information Services",
"Programming Language :: Python :: 3",
# Note: this is our standard license for open projects.
"License :: OSI Approved :: MIT License",
This should be suitable for most of our Python packages. It assumes there is a
`README.md` file which documents the product and a `requirements.txt` file which
specifies the dependencies.
### Add PyPI jobs to GitLab CI
We make use of GitLab CI to publish packages. We have a standard template
available which may be included into the project's `.gitlab-ci.yml` file as
# .gitlab-ci.yml contains configuration for GitLab CI jobs.
# ... other includes
# Support uploading to PyPI
- project: 'uis/devops/continuous-delivery/ci-templates'
file: '/pypi-release.yml'
### Create a temporary test PyPI API token
The PyPI publication jobs won't be added to pipelines unless the appropriate
`PYPY_API_TOKEN` or `TEST_PYPI_API_TOKEN` variables are defined in the GitLab CI
configuration and so the next step is to create an API token.
PyPI allows API tokens to have "scopes" limiting them to certain projects. This
presents us with a bit of a chicken-and-egg problem: we want to use an API token
limited in scope to the project we're releasing but we can't *select* that scope
until we've published the project. We'll have to do a bit of token gymnastics to
get around this.
Sign into the [test PyPI instance](https://test.pypi.org/) using the
`uis-devops-bot` account credentials. [Create a new API
token](https://test.pypi.org/manage/account/token/) with the following settings:
* Name: **Temporary all scopes token to publish [PACKAGE NAME]**
* Scope: **Entire account (all projects)**
Make sure to copy the API token somewhere safe since it will only be shown to
you once.
### Add the temporary test PyPI API token as a GitLab CI variable
In the CI/CD settings for the target project on GitLab, add a new variable:
* Value: _paste in API token_
* Protect variable: **yes**
* Mask variable: **yes**
We have configured the variable as being *protected*. This means that it
will only be available to CI jobs running on [protected
By default only "master" is a protected branch in our standard project
### Make the first release
Trigger a CI/CD pipeline run for the release branch. (This can be done via the
**Run Pipeline** button on the CI/CD pipelines page for the GitLab project.) A
new [manual job](https://docs.gitlab.com/ee/ci/yaml/#whenmanual) is created
called `test-pypi-release`.
Visit the pipeline page for the triggered pipeline and click the "play" button
next to `test-pypi-release`. Confirm that an initial release has been made to
the Test PyPI by looking in the log for the job.
If there is no `test-pypi-release` job, make sure that the branch the
pipeline is running on is protected.
### Replace the API token with a scoped token
On the [account management page](https://test.pypi.org/manage/account/) for test
PyPI, remove the temporary token created above and [create a new
* Name: **GitLab CI for [PACKAGE NAME]**
* Scope: *Select the appropriate project*
Copy the new token and replace the value for the `TEST_PYPI_API_TOKEN` in the
GitLab CI/CD settings.
### Test the new token
Trigger a pipeline for the release branch and start the `test-pypi-release` job
as before. The publication will fail because PyPI does not allow uploading new
packages with the same version number but you shouldn't get a failure due to bad
### Repeat for the production PyPI
Once you are happy publication is working well for the test PyPI, repeat the
instructions for the [production PyPI instance](https://pypi.org/). The steps
are the same except that tokens should be created in the main PyPI instance,
the GitLab CI/CD variable should be `PYPY_API_TOKEN` and the job is called
## Publishing a previously published package
This is far simpler because some kind soul will already have performed the
configuration above.
### Bump the version number if necessary
Before you publish make sure that the version of the package as set in
`setup.py` is greater than the version currently published on PyPI. We use
[semver](https://semver.org/) when versioning our packages.
Python packages support adding "devN" to a version to indicate a development
release which should not be installed unless explicitly asked for. As such,
if the most recent published version is "1.2.3", you can publish "1.2.3dev4"
without it becoming the automatically installed version.
### Publish to the test PyPI
On the GitLab project page, click on the "pipeline status" badge to get to the
most recently run pipeline on "master". There should be a manual job waiting for
you there called `test-pypi-release`. Click the "play" button and check the job
logs to see if the package was successfully published.
### Publish to production PyPI
If the test publication succeeds, click the "play" button next to the
`pypi-release` job in the same pipeline. This should publish the package to the
production PyPI site.
......@@ -20,6 +20,7 @@ nav:
- workflow/flows.md
- workflow/merge-requests.md
- workflow/onboarding.md
- workflow/pypi.md
- 'Best Practice':
- best-practice/index.md
- best-practice/git.md
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment