FAQ | This is a LIVE service | Changelog

Skip to content

auto-generate API clients, package them, publish them and publish docs to GitLab pages

Dr Rich Wareham requested to merge issue-23-generate-api-clients into main

Part of #23 (closed). Lays groundwork for uis/devops/continuous-delivery/ci-templates#93 (closed) and #22 (closed).

This MR extends the existing schema generation template to support auto-generation of OpenAPI clients and their documentation. This MR:

How to review

  • Read the "Small fixes" section below to see incidental fixes made by this MR.
  • Read .gitlab/openapi-generator.md which is intended to form the kernel of the Guidebook documentation.
  • Read .gitlab/openapi-generator-local.gitlab-ci.yml and .gitlab/docs-local.gitlab-ci.yml as they give you an idea of the "special" config for this repo.
  • Read mkdocs.yaml in the root to give you an idea of the special docs config for this repo. Most of docs/ itself is intended to be moved to the boilerplate and has already been through one round of review in the Regent House Ballots application. It is a straightforward copy-paste and ideally will survive unchanged into the copier boilerplate.
  • Read .gitlab/openapi-generator.gitlab-ci.yml to see how the sausage is made.

Individual commits provide wider context but summaries are provided below.

Work intended for boilerplates and/or CI template repo

The config intending to stay in this repository is that in mkdocs.yaml in the root and the .gitlab/*-local.gitlab-ci.yml files. The OpenAPI generator CI config is intended to form part of the Common CI pipeline (and as such does nothing unless enabled by the presence of a file or explicitly via CI variables). We do some light customisation of the mkdocs template to deal with incorporating the generated client documentation.

The majority of docs/ is intended to make its way into the boilerplate. Most of it is relatively uninteresting but docs/scripts/gen-reference-nav.py is intended to be generic and is currently present in both this MR and the Regent House Ballots webapp. Should docs/ survive review I intend to move a fair chunk of it into the copier template. This should have little to no effect on this project once merged but will enable other projects to have automatically generated documentation in future.

Small fixes

Some small fixes were needed before starting work on this MR. They are present as separate commits but, in brief:

  • add a maximum line length to .editorconfig for markdown files to appease my linter.
  • bump the flake8 version to allow it to parse the more modern f-string idiom we use.
  • adding missing __init__.py files for management command module which broke documentation generation
  • add a missing LICENSE file referenced from the README
  • fix incorrect indenting in some docstrings
  • adding type annotations consistent with the docstrings

All of the above docstring warts cause documentation generation to fail because consistency between docstrings and type annotations is now enforced.

Versioning

The behaviour of the generic OpenAPI generator template is to base the API client version on the schema version. This causes an issue when building and publishing clients since both npm and PyPI don't allow you to "re-publish" clients with the same version. As such we add an additional requirement to the basic schema job rule that the schema must actually have changed in order to regenerate clients. This still has the small wart that you'll only see failure to publish because you forgot to bump the version spec on runs to main but for this particular project, that's not an issue since we override schema generation to make use of the release version and that will always be different when commits hit main.

Client generation

In order to have some confidence that our approach is language-neutral, we generate two clients: one TypeScript and one Python. The generation jobs are based on generic npm and python templates and individual clients can be disabled via ..._DISABLED variables if desired by downstream users.

We choose sensible defaults for package names, etc but these can be overridden globally or per-generator. We demonstrate this feature in this MR since the default package name is the GitLab project name and for this project the name, api, is not terribly descriptive. Additionally Python module names can't contain - and so we additionally override the Python package name.

Generated clients are written to $OPENAPI_GENERATOR_SOURCE_ARTIFACT_DIR and are available to later CI jobs should they be needed.

Packaging

Once generated, clients are packaged. Again we create generic npm and Python packaging jobs and specialise them for our particular clients. The build artefacts are written to $OPENAPI_GENERATOR_PACKAGE_ARTIFACT_DIR. We would add those artefacts to the release-it configuration but, currently, release-it is unable to add binary assets to releases.

Once release-it is fixed, adding the assets is a case of adding a single line to .release-it.json to specify the package artefact locations.

TypeScript clients are packaged as a tarball which is suitable for passing to npm install and/or npm publish and Python clients generate source distribution tarballs and wheels.

Publishing

On commits to main, the packaged client artefacts are published to the GitLab package registry. You can see an example of this at https://gitlab.developers.cam.ac.uk/uis/devops/experiments/rjw57/misc/-/packages.

image

We use semver-compatible "build metadata" in the version number to additionally encode both the CI job id which generated the package and the git commit SHA. This is good practice in general for being able to tie package artefacts back to the processes which created them.

Client Documentation

We use language-appropriate documentation generators to document each generated client and use redoc to document the API schema itself. These are all created via documentation jobs and stored as job artefacts. The default template doesn't do anything with them but this MR shows a potential publication pathway.

Project Documentation

Our common CI pipeline supports mkdocs-based documentation natively and so we make use of that to generate project documentation which includes the API client documentation. This is a "MVP" effort at the moment. It's likely to need some grooming as time goes on. The mkdocs config is based on one intended to end up in the webapp boilerplate and this MR provides additional baking to that end.

On commits to main the generated documentation is published via GitLab pages.

The documentation generated from this MR may be previewed at https://uis.uniofcam.dev/-/devops/iam/activate-account/api/-/jobs/1916064/artifacts/public/index.html

For example, here is some of the generated API docs:

image

Following the "API Clients" link leads you to a list of API client documentation. For example, for TypeScript:

image

and for Python:

image

The "REST API Reference" link leads you to the API documentation generated by redoc. For example:

image

Further work

The generated project documentation is intentionally sparse since that's not the main focus of this MR. I've added just enough to actually make the documentation a) exist and b) link to the API client documentation but more work is needed. In particular there is no information on how to install any of the generated clients.

Edited by Dr Rich Wareham

Merge request reports