docs: Add all finalized tech lead forum standards

docs/libraries-tools/gcp-terraform/index.md

@@ -10,7 +10,7 @@ Our adopted & recommended tools & services for Google Cloud Platform (GCP) and T
 | [Cloud Run](https://cloud.google.com/run)                       | Serverless container platform for stateless applications | [Reference](../../reference/cloud-platform/index.md), [Migration](../../explanations/gcp-cloud-run-app-module.md) |
 | [Cloud SQL](https://cloud.google.com/sql)                       | Fully managed relational database service                | [Backups](../../reference/cloud-platform/backups.md)                                                              |
 | [Cloud Storage](https://cloud.google.com/storage)               | Object storage service                                   | [Reference](../../reference/cloud-platform/index.md)                                                              |
-| [Cloud Load Balancing](https://cloud.google.com/load-balancing) | Scalable load distribution service                       | [Architecture](../../reference/standard-architecture-diagrams.md)                                                 |
+| [Cloud Load Balancing](https://cloud.google.com/load-balancing) | Scalable load distribution service                       | [Architecture](../../standards-and-compliance/standard-architecture-diagrams.md)                                  |
 | [Cloud DNS](https://cloud.google.com/dns)                       | Managed DNS service                                      | [DNS Guide](../../reference/cloud-platform/dns.md)                                                                |
 | [Secret Manager](https://cloud.google.com/secret-manager)       | Secure secret storage service                            | [CI Access](../../howtos/gke-gitlab-runners/access-secrets-in-ci-jobs-using-impersonation.md)                     |
 | [Cloud Monitoring](https://cloud.google.com/monitoring)         | Infrastructure and application monitoring service        | [Reference](../../reference/cloud-platform/index.md)                                                              |

docs/reference/tech-lead-forum.md

@@ -10,7 +10,7 @@ title: The Tech Lead Forum
 
 This page lists the purpose, the composition and the processes for the Tech Lead Forum. An
 explanation of why the Tech Lead Forum exists and how it helps us is available on [a separate
-page](../explanations/tech-lead-forum.md).
+page](../standards-and-compliance/tech-lead-forum.md).
 
 ## Purpose
 

docs/standards-and-compliance/devsecops-cicd.md

+---
+title: Continuous Integration and Continuous Delivery (CI/CD)
+---
+
+# DevOps standards for Continuous Integration and Continuous Delivery (CI/CD)
+
+## Introduction
+
+This document provides an overview of the University of Cambridge Information Services’ (UIS’)
+Development and Operations (DevOps) Division approach to implementing Continuous Integration and
+Continuous Delivery (CI/CD) as per ISO standard ISO/IEC/IEEE 32675:2022(E) “Information technology —
+DevOps — Building reliable and secure systems including application build, package and deployment”.
+
+CI/CD is the automation of most or all of the manual processes traditionally required to get new
+code deployed to production. It encompasses the build, review, test and deployment processes. Since
+we provision infrastructure as code, this extends to infrastructure provisioning. With a CI/CD
+pipeline, development teams can make changes to code that are then automatically tested and pushed
+out for delivery and deployment. Automation makes processes predictable, reviewable, and repeatable
+so that there is less opportunity for error from human intervention.
+
+In this document we consider CI/CD broadly to encompass processes which a) are automated, b) have
+outputs which relate to the DevOps development lifecycle and c) are triggered by actions by
+engineers or on a schedule. Notably absent from this are scheduled processes which run as part of
+services themselves such as scheduled data imports, syncs or other batch processes.
+
+Where CI/CD processes are programming language specific, the provision of standard CI/CD templates
+may not be the same between languages. It is desirable to close feature gaps between languages where
+they exist, for example, by generating templates that are compatible with multiple languages.
+
+This document describes CI/CD standards at a high-level. Specifics and implementation details may be
+found elsewhere. For
+example, [our guidebook has a page on how to add CI/CD to a Python project](../howtos/add-common-pipeline-to-python.md)
+and [a page on configuring release automation via CI/CD](../howtos/enable-automated-gitlab-releases.md).
+Our existing [web application](https://gitlab.developers.cam.ac.uk/uis/devops/webapp-boilerplate/)
+and [terraform deployment](https://gitlab.developers.cam.ac.uk/uis/devops/gcp-deploy-boilerplate)
+boilerplates already include the standards described below.
+
+CI/CD automation is performed via GitLab pipelines. These standards are either implemented in the
+shared
+DevOps [CI template configurations](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/)
+or use out of the box solutions.
+
+## Applicability
+
+This standard should be implemented for all services that the university develops, whether
+internally, or through third parties. It is particularly relevant to services that handle sensitive
+data or support important processes, where a security breach could have significant negative
+impacts.
+
+Exceptions can be submitted following the exemption process within the Systems Management Policy
+informing and seeking approval of the Tech Leads Forum first
+
+## Terminology
+
+A **job** is an automated process which usually takes as its input a specific commit of the code and
+some configuration variables. Jobs may generate **artefacts** which are either files shared
+subsequently with other jobs or files which are published to some external registry. Jobs may have
+[**intended side-effects**](https://en.wikipedia.org/wiki/Side_effect_\(computer_science\)) such as
+releasing a new version of some software or provisioning some infrastructure.
+
+Jobs are collected into **pipelines**. Within a pipeline, jobs may depend on one another for
+artefacts or side-effects. The configuration of jobs and pipelines lives within the git repository
+of a project, usually in a `.gitlab-ci.yml` file.
+
+Pipelines are **triggered** from multiple sources. Most commonly we use **push**\-triggered
+pipelines which trigger when new commits are pushed to a repository, **tag**\-triggered pipelines
+which trigger when commits are tagged and **scheduled** pipelines which run according to a defined
+schedule.
+
+Jobs within pipelines may indicate that they deploy to **environments**. The most recent jobs which
+have deployed to an environment can be seen in the “Environments” tab in GitLab.
+
+<figure markdown>
+  ![](./images/devsecops-cicd-7.png)
+  <figcaption markdown>
+    Figure taken from our
+Guidebook [tutorial on creating a Python package](../tutorials/creating-a-python-package/).
+  </figcaption>
+</figure>
+
+## Cloud Platform-as-a-Service
+
+The following sections detail our standard CI pipeline workflows. While the concepts are generic,
+the implementation of our custom CI templates is specifically designed for use with our standard
+Platform-as-a-Service (PaaS) based on Google Cloud and follows our published standards on
+Development and Deployment. This is so that we can ensure certain tasks which may require additional
+thought, such as authentication to external systems, are implemented in a consistent, approved and
+secure way. As such, there are some core cloud platform prerequisites which must be in place before
+using many of the workflows below, they are:
+
+- A folder/product needs to be created first on Google Cloud using
+  the [gcp-product-factory](https://gitlab.developers.cam.ac.uk/uis/devops/infra/gcp-product-factory).
+  This will create the space in our PaaS to host your service and will ensure that all the required
+  pre-configuration is securely deployed following our standards.
+- The GitLab projects which will store source code and run the CI/CD pipelines must be deployed via
+  the [gitlab-project-factory](https://gitlab.developers.cam.ac.uk/uis/devops/infra/gitlab-project-factory).
+  Existing projects can be imported.
+- A GitLab Runner must have been deployed for the GitLab projects in question via
+  the [gitlab-runner-infrastructure](https://gitlab.developers.cam.ac.uk/uis/devops/devhub/gitlab-runner-infrastructure)
+  Terraform configuration. This ensures that the GitLab runner required to execute CI and CD jobs
+  has the correct configuration and is secure.
+
+For a more detailed explanation on our Google Cloud standards see
+the [Cloud Platform](https://guidebook.devops.uis.cam.ac.uk/en/latest/reference/cloud-platform/)
+section in the Guidebook.
+
+## GitLab Auto-DevOps
+
+Pipelines should use [GitLab’s Auto-DevOps feature](https://docs.gitlab.com/ee/topics/autodevops/).
+This provides a number of standard jobs for tasks such as container image builds, dependency
+scanning, secret detection, and code quality.
+
+Enabling Auto-DevOps can be as simple
+as [including the Auto-DevOps template](https://docs.gitlab.com/ee/topics/autodevops/customize.html#customize-gitlab-ciyml)
+in a project’s .gitlab-ci.yml file. However, in practice we have embedded the Auto-DevOps template
+into our own CI templates to allow us to extend the Auto-DevOps pipeline, adding additional
+functionality where required. For example, our
+standard [web application boilerplate](https://gitlab.developers.cam.ac.uk/uis/devops/webapp-boilerplate/-/blob/master/%7B%7B%20cookiecutter.project_slug%20%7D%7D/.gitlab/webapp.gitlab-ci.yml?ref_type=heads#L3)
+uses
+our [common-pipeline.yml](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/auto-devops/common-pipeline.yml?ref_type=heads),
+which should be the default template used in any of our Python projects.
+
+Our CI templates can be found in the
+dedicated [ci-templates GitLab project](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/).
+
+## Stages
+
+CI/CD pipelines have multiple stages corresponding to different classes of work. The following are
+jobs you should employ in CI/CD pipelines, grouped by stage. As mentioned above, some of the jobs
+described below ship
+with [GitLab’s Auto DevOps feature](https://docs.gitlab.com/ee/topics/autodevops/) and some have
+been created by UIS DevOps.
+
+### Build and package
+
+For web application or library code, we build appropriate *build artefacts*. For web applications,
+this will be a container image built using the Auto-DevOps Build job. For libraries this is
+generally some language-specific package intended for upload to package registries such as maven,
+PyPI or npm. In this case we have to implement our own build job. For PyPI, this is implemented
+using our
+standard [python-publish template](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/auto-devops/python-publish.yml?ref_type=heads#L11).
+
+### Analyse
+
+We employ a number of static analysis jobs which generate *report artefacts*. As a baseline, we must
+include all the Auto-DevOps Dependency Scanning as well as the other Auto-DevOps analysis jobs such
+as Container Scanning, Code Quality, Secret Detection, and Static Application Security Testing (
+SAST).
+
+The output from these analysis jobs is provided back to GitLab which allows us to rapidly scan for
+projects using out-of-date dependencies or ones whose container images contain libraries with known
+vulnerabilities.
+
+GitLab security policies run this analysis step for the `main` branch regularly so that we may scan
+for new vulnerabilities as they arise.
+
+### Test and lint
+
+“[Linting](https://en.wikipedia.org/wiki/Lint_\(software\))” is a term of art used in Software
+Development to refer to automated code formatting checks. These checks do not check functional
+correctness *per se* but rather look for common patterns which, although still allowing the code to
+function correctly, are considered detrimental to readability or maintainability. Recently it has
+become common to enforce an opinionated code formatting style for reasons of consistency and for
+that code formatting style to be amenable for automatic application. Below we’ll use “code format
+checking” to mean the use of both linting and more opinionated code style tooling.
+
+In the test and lint stage there will be some form of automated testing and code style review. It is
+outside this paper’s scope to cover test standards, which is covered in a separate paper. For our
+Python projects, use
+our [python-tox template](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/auto-devops/python-tox.yml?ref_type=heads)
+to run tests using [tox](https://tox.wiki/en/4.11.4/), and for our Terraform repositories use the
+jobs in the test stage of
+the [terraform-pipeline template](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/auto-devops/terraform-pipeline.yml?ref_type=heads).
+
+For projects with unit-tests it will generate a *code-coverage artefact* which indicates which code
+the unit-tests actually tested. This is used later in the review section.
+
+<figure markdown>
+  ![](./images/devsecops-cicd-4.png)
+  <figcaption markdown>
+    Testing jobs usually comprise the majority of pipelines triggered by commits to the code.
+  </figcaption>
+</figure>
+
+For pipelines which are triggered by tags, we will also perform some basic correctness checks such
+as ensuring that the version of the software in the packaging metadata matches the name of the tag.
+This is to enforce that a tag named 1.2.3 does actually correspond with version 1.2.3 of the
+software.
+
+We also use [pre-commit hooks](https://pre-commit.com/index.html) to specify which code formatting
+tools, and configuration, to use. Use
+our [pre-commit template](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/auto-devops/pre-commit.yml?ref_type=heads)
+to enforce pre-commit checks in CI/CD. The use of pre-commit allows developers to configure tools to
+run locally as well when committing changes. This helps lower the latency of the feedback loop for
+what may be viewed as “trivial” code formatting changes. This also helps ensure that reviewers stay
+focused on content rather than style.
+
+### Review
+
+GitLab Merge Requests (MRs) aggregate a number of reports generated by CI/CD jobs. Exactly which
+jobs are run depends on the nature of the repository. For web application and library code, the code
+coverage artefact will be used to highlight the changed portions of code depending on whether the
+tests exercised them or not. Similarly, a test report will be attached to the MR indicating which
+tests ran and which passed. MRs cannot ordinarily be merged unless all tests pass. When a container
+image is built, tests are run *within* the container to provide some confidence that the packaging
+is successful.
+
+Code format checking is treated as a test in its own right and so formatting issues will need to be
+addressed or manually skipped for MRs to be merged.
+
+While we currently perform dependency and security vulnerability analysis for MRs, we do not prevent
+merging if new vulnerabilities are determined. GitLab provides mechanisms to define security
+policies surrounding vulnerability triaging and reviewing which we are in the process of adopting
+and will be covered in another paper.
+
+<figure markdown>
+  ![](./images/devsecops-cicd-9.png)
+  <figcaption markdown>
+    Bill-of-materials analysis for a project in GitLab. The analysis for this page is performed as a
+CI/CD job.
+  </figcaption>
+</figure>
+
+### Release and publish
+
+Jobs in this stage are performed only on commits which have been merged into the default branch.
+Even so, we run the test and lint jobs once more and do not proceed until they pass.
+
+Use our standard release automation workflow for creating and tagging new releases. Details of this
+workflow
+are [available in the guidebook](https://guidebook.devops.uis.cam.ac.uk/en/latest/explanations/gitlab-release-automation/)
+but in brief, we have two forms of release automation:
+
+Variant A involves jobs triggered when new commits land in the default branch. These jobs curate a
+`CHANGELOG.md` file and increment the version number specified in the packaging configuration.
+Version numbers are incremented in line with semantic versioning informed
+by [conventional commit messages](https://guidebook.devops.uis.cam.ac.uk/en/latest/explanations/git/#commits).
+In this case, use of
+the [commitlint job](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/commitlint.yml?ref_type=heads)
+which will enforce conventional commit naming schemes. Once the `CHANGELOG.md` file and versioning
+metadata are updated, a new tag and GitLab release is created named after the version.
+
+Variant B similarly involves jobs triggered when new commits land in the default branch. This time
+changes to the version number and `CHANGELOG.md` file are kept in a separate MR. Merging this MR
+effectively causes a new tag and GitLab release to be created. This variant is suitable for products
+where new releases should not be created for each change which lands in the default branch.
+
+<figure markdown>
+  ![](./images/devsecops-cicd-1.png)
+  <figcaption markdown>
+    Example MR managed by variant B of the automated release process.
+  </figcaption>
+</figure>
+
+<figure markdown>
+  ![](./images/devsecops-cicd-8.png)
+  <figcaption markdown>
+    The GitLab release and automated changelog generated when merged.
+  </figcaption>
+</figure>
+
+No matter how the git tag corresponding to a release is made, once created a pipeline is triggered
+to publish the build artefact for that release. For libraries this will involve publishing the
+artefact to a package registry appropriate for the language. This will be a public registry for Open
+Source libraries or the private GitLab registry for private or internal libraries.
+
+For web applications, publication involves pushing the container image to the Google Cloud Artifact
+Registry tagged with the version, ready for deployment. Use
+our [artifact-registry template](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/auto-devops/artifact-registry.yml?ref_type=heads)
+for this task. This template handles authentication to both the GitLab Container Registry and the
+Google Cloud Artifact Registry in a secure and approved manner.
+
+<figure markdown>
+  ![](./images/devsecops-cicd-3.png)
+  <figcaption markdown>
+    CI/CD pipeline triggered by a new git tag which builds, tests and publishes a container image to
+the Google Artifact Registry.
+  </figcaption>
+</figure>
+
+<figure markdown>
+  ![](./images/devsecops-cicd-5.png)
+  <figcaption markdown>
+    The container images pushed to the Google Artifact Registrty ready for deployment.
+  </figcaption>
+</figure>
+
+### Deploy
+
+Deployment of services and provision of infrastructure is done via an infrastructure as code using
+Terraform. Terraform code is code like any other and so many of the CI/CD jobs described above are
+equally applicable.
+
+Use
+our [terraform-pipeline CI template](https://gitlab.developers.cam.ac.uk/uis/devops/continuous-delivery/ci-templates/-/blob/master/auto-devops/terraform-pipeline.yml?ref_type=heads)
+for Terraform deployments. This template is designed explicitly for Terraform and runs test and
+code-format checking jobs as well as “plan” and “apply” jobs for our standard development, staging
+and production environments.
+
+These jobs are triggered when new commits are pushed to the repository. The “plan” jobs allow manual
+review of infrastructure changes in advance of “apply”. Any commit may be deployed to the
+development environment but, ordinarily, only commits which land in the default branch should
+trigger deployments to staging or production environments.
+
+For commits which are merged into the default branch, the staging environment is, unless configured
+otherwise, deployed automatically. This lets us treat the staging environment as the most
+“up-to-date” infrastructure and ensure that the default branch is always deployable. Jobs are
+created for deployment to the development and production environments but they require manual
+approval to proceed. Manual approval for deployment to the production and development environments
+can be performed in the Pipelines view by pressing the “play button” next to the appropriate
+environment to trigger the deployment.
+
+<figure markdown>
+  ![](./images/devsecops-cicd-2.png)
+  <figcaption markdown>
+    The container images pushed to the Google Artifact Registrty ready for deployment.
+  </figcaption>
+</figure>
+
+GitLab also has an Environments view, which can be very useful to see at a glance which version is
+currently deployed to each environment. However, the “play” and “stop” buttons in this view should
+not be used as our Terraform pipeline jobs have not yet been configured with the required
+logic/functions and, therefore, jobs triggered from the Environments view can perform unintended or
+confusing actions. Triggering manual pipeline jobs must be done via the Pipelines view, as described
+above until support has been added.
+
+<figure markdown>
+  ![](./images/devsecops-cicd-6.png)
+  <figcaption markdown>
+    The “Environments” page for a product in GitLab showing which version is currently deployed to the
+staging environment.
+  </figcaption>
+</figure>