Add detached pipeline and deployment workflow templates

Adds a detached pipeline workflow template which runs the CI flow for both merge requests and branches without merge requests. An additional deployment template is added to implement the default webapp deployment flow.

The detached pipeline template in this MR is a copy of gitlab-ci/auto-devops-for-detached-pipelines.yml in MR uis/devops/uga/smi!221, so see that MR for a test pipeline based on the detached pipeline workflow template.

The deployment template in this MR is basically a copy of the SMI deploy CI config file with the webapp URLs converted to variables, and some optimisation of the rules.

Closes #5 (closed)

Closes #6 (closed)

changed the description

How do we deal with projects with multiple webapps? E.g. https://gitlab.developers.cam.ac.uk/uis/devops/uga/deploy/-/merge_requests/19/diffs

As long as all the deployment jobs extend .cloud-run-deploy then adding extra jobs for multiple webapps should require minimal configuration. As has been done in uis/devops/uga/deploy!19, SERVICE_PREFIX should be added as a variable to .cloud-run-deploy that permits it to be configured for multiple webapps.

assigned to @amc203

While GitLab CI documentation confirms that there is no dependency scanner for Terraform, the jobs will be skipped, so I think is safe not to include this line. Trying to be future proof and not modify autodevops when not necessary.

We need to start versioning our CI templates to avoid having the same problems we have with tox-tests.yml which we can't update because it will break some pipelines. I think for this we should follow the same solution proposed in #7

I think DAST should be executed to dev and staging deployments.

Some SAST needs to be enabled for the deployment repos, reading GitLab documentation we can see that we should be executing at least, Gitleaks, TruffleHog, Kubesec. As with others, I believe that GitLab SAST Template should be sufficiently clever to skip those that do not apply to the repo, and thus, this line shouldn't be necessary.

I think having all these exceptions is showing that probably we are not doing it properly to make it work with AutoDevOps. I believe this is because we are executing CD jobs in the Infrastructure as code (IAC) repository. IAC still requires Gitlab CI pipelines as our infrastructure is code, so it needs testing, SAST, etc.

If we look at this CI template there is nothing in there is nothing in here that uses any code of the IAC repo. When deploying, it does not even uses the build step, or any terraform code.

The only thing that we use from the repo are the secrets. I wonder then if we are architecting our repos wrong and we need:

• WebApp repo. Webapp code. Normal AutoDevOps but with review apps. Maybe review apps deployment should used the CD repo defined below?
• Infra as code repo. Terraform code for the infrastructure. Normal AutoDevOps for checking dependencies, secrets, test, etc. Deployment happens only once (or rarely when the infra definition changes). No need for auto-deployment of infra.
• CD repo. Will only contain gitlab-ci.yml file as code. That file will only import this template. Deployment secrets will only be stored as CI env vars in this repo. Will have a button to deploy to production, after a a staging deployment has happened DAST and E2E tests have been executed against staging (probably calling the webapp repo which will contain the e2e test code).

I think this section is too fragile and won't cope with future GitLab CI upgrades. What are we trying to achieve with this?

The purpose of this section (and indeed the rest of the file) is to change our workflow. The GitLab default is the rule if: '$CI_COMMIT_TAG || $CI_COMMIT_BRANCH' (which is the "branch pipeline" workflow), whereas using if: $CI_MERGE_REQUEST_IID || $CI_COMMIT_TAG || $CI_COMMIT_BRANCH gives us a merge request + branch workflow (a variant of the "merge request pipeline" workflow).

I don't think there's a way to change the Auto-DevOps workflow other than to change the rules for every single job. This situation might improve when this GitLab issue is implemented to permit variables to be defined based on other variables (scheduled for GitLab 13.5).

#9 (closed) implements the needs dependency of the test job on the build job for the Tox-based test pipeline. Possibly we don't want this in the basic detached pipeline template?

tox tests been looked at #7

Closing without merging as detached pipelines will no longer be used. A new merge request will be created with just the deployment template (for #6 (closed)).

closed

changed the description