Merge branch 'get-gcp-secrets-fragment' into 'master'

Add a generic fragment to retrieve Google Secret Manager secrets.

Closes #60

See merge request !86

CHANGELOG.md

@@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [4.1.0] - 2024-05-16
+
+### Added
+
+- A generic `get-gcp-secrets` fragment to retrieve one or more Google Secret
+  Manager secrets in a CI job.
+
 ## [4.0.0] - 2024-04-05
 
 ### Changed

auto-devops/release-it.yml

-# This template enables automated release management using the release-it tool. 
+# This template enables automated release management using the release-it tool.
+
+include:
+  - local: /fragments/get-gcp-secrets.yml
 
 .release-base:
   image:
@@ -6,50 +9,34 @@
   variables:
     GIT_STRATEGY: clone
     GIT_DEPTH: 0
-  before_script: |
-    MAGENTA="\e[35m"
-    CLEAR="\e[0m"
-
-    log() {
-        echo -e "${MAGENTA}${1}${CLEAR}"
-    }
-
-    # By default, GitLab performs a shallow clone of the target repository and checks out a detached commit rather than
-    # the branch itself. We want to be able to search the full history so we need to do the following.
-    git fetch origin
-    git checkout "$CI_COMMIT_REF_NAME"
-
-    log "Retrieving an access token for the default service account of the runner pod..."
-    DEFAULT_TOKEN=$(
-      curl --fail-with-body -s -S -H "Metadata-Flavor: Google" \
-        "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
-      | jq -r ".access_token"
-    )
-
-    log "Generating an access token for the $GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT service account..."
-    ACCESS_TOKEN=$(
-      curl --fail-with-body -s -S -X POST \
-        -H "Authorization: Bearer $DEFAULT_TOKEN" \
-        -H "Content-Type: application/json; charset=utf-8" \
-        "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT/@/%40}:generateAccessToken" \
-        -d '{"scope": ["https://www.googleapis.com/auth/cloud-platform"]}' \
-      | jq -r ".accessToken"
-    )
-
-    log "Retrieving the GitLab bot access token from Google Secret Manager..."
-    export GITLAB_TOKEN=$(
-      curl --fail-with-body -s -S -H "Authorization: Bearer $ACCESS_TOKEN" \
-        -H "Content-Type: application/json" \
-        "https://secretmanager.googleapis.com/v1/$GITLAB_ACCESS_TOKEN_SECRET_ID/versions/latest:access" \
-      | jq -r ".payload.data" | base64 -d
-    )
+  before_script:
+    # If our standard GITLAB_ACCESS_TOKEN_SECRET_ID variable is set we re-export it to a variable with the correct
+    # name/value for the get-gcp-secrets helper script.
+    - |
+      if [[ -n "$GITLAB_ACCESS_TOKEN_SECRET_ID" ]]; then
+        export GET_GCP_SECRET_GITLAB_TOKEN="$GITLAB_ACCESS_TOKEN_SECRET_ID/versions/latest"
+      fi
+    # If our standard GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT variable is set we re-export it as the generic
+    # GOOGLE_IMPERSONATE_SERVICE_ACCOUNT variable that the get-gcp-secrets helper script expects.
+    - |
+      if [[ -n "$GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT" ]]; then
+        export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT="$GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT"
+      fi
+    # Call the get-gcp-secrets helper script to retrieve the $GET_GCP_SECRET_GITLAB_TOKEN secret value and export it as
+    # GITLAB_TOKEN for the release-it tool to access.
+    - !reference [.get-gcp-secrets, before_script]
+    - |
+      # By default, GitLab performs a shallow clone of the target repository and checks out a detached commit rather than
+      # the branch itself. We want to be able to search the full history so we need to do the following.
+      git fetch origin
+      git checkout "$CI_COMMIT_REF_NAME"
 
-    log "Configuring git to use the GitLab bot account..."
-    git config user.email "$GITLAB_ACCESS_TOKEN_EMAIL"
-    git config user.name "$GITLAB_ACCESS_TOKEN_NAME"
+      echo "Configuring git to use the GitLab bot account..."
+      git config user.email "${GITLAB_ACCESS_TOKEN_EMAIL:-release-it-bot@noreply.gitlab.developers.cam.ac.uk}"
+      git config user.name "${GITLAB_ACCESS_TOKEN_NAME:-release-it-bot}"
 
-    log "Configuring the git remote url..."
-    git remote set-url origin "https://token:$GITLAB_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
+      echo "Configuring the git remote url..."
+      git remote set-url origin "https://token:$GITLAB_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
   tags:
     - $GKE_RUNNER_TAG
 

fragments/get-gcp-secrets.md

+# GCP Secrets Helper
+
+Accessing secrets securely in CI/CD pipelines is notoriously tricky. This
+fragment provides some reusable code to make the process of retrieving one
+or more Google Secret Manager secrets easier.
+
+## Prerequisites
+
+To use this fragment in your CI jobs you will need to ensure that the following
+prerequisites are met.
+
+- You'll need to ensure that `bash`, `curl`, and `jq` are installed in your
+  jobs' image and that your job is running in a `bash` shell.
+- The CI runner executing your job must be running in the context of a Google
+  Cloud identity. The identity must either have the `secrets.secretAccessor`
+  role granted for all secrets to be retrieved, or have the ability to
+  impersonate a service account which does.
+
+## How-to
+
+The following steps will retrieve the `latest` version of the Google Secret
+Manager secret named `my-secret` from the Google Cloud project named
+`my-project` and export the secret value as an environment variable named
+`MY_SECRET`.
+
+1. First, in your `.gitlab-ci.yml` file include the `get-gcp-secrets.yml`
+    template.
+
+   ```yaml
+   include:
+     - project: uis/devops/continuous-delivery/ci-templates
+       file: /fragments/get-gcp-secrets.yml
+       ref: <specify version here>
+   ```
+
+1. Next, define a job with the variable `GET_GCP_SECRET_MY_SECRET:
+    projects/my-project/secrets/my-secret/versions/latest`.
+
+   ```yaml
+   my-job:
+     stage: build
+     image: ubuntu:24.04
+     variables:
+       GET_GCP_SECRET_MY_SECRET: projects/my-project/secrets/my-secret/versions/latest
+   ```
+
+1. Specify the `$GKE_RUNNER_TAG` job tag. When working on UIS DevOps projects,
+    you'll usually need to run your CI jobs on one of our Google Kubernetes
+    Engine hosted runners to be able to use this helper. To do that, add the
+    magic `$GKE_RUNNER_TAG` variable to the list of tags for your job.
+
+   ```yaml
+   my-job:
+     stage: build
+     image: ubuntu:24.04
+     variables:
+       GET_GCP_SECRET_MY_SECRET: projects/my-project/secrets/my-secret/versions/latest
+     tags:
+       - $GKE_RUNNER_TAG
+   ```
+
+1. Add a reference to the included hidden `.get-gcp-secrets` job's
+    `before_script` block in your job's `before_script`.
+  
+   ```yaml
+   my-job:
+     stage: build
+     image: ubuntu:24.04
+     variables:
+       GET_GCP_SECRET_MY_SECRET: projects/my-project/secrets/my-secret/versions/latest
+     before_script:
+       - apt-get update && apt-get install -y curl jq # Required as this example uses the ubuntu:24.04 base image.
+       - !reference [.get-gcp-secrets, before_script]
+   ```
+
+1. The `before_script` referenced in `.get-gcp-secrets` will retrieve the value
+    of the secret in the `projects/my-project/secrets/my-secret/versions/latest`
+    secret object and export it as an environment variable named `MY_SECRET`.
+    This variable can then be referenced in the `script` block of your job in the
+    usual way.
+
+   ```yaml
+   my-job:
+     stage: build
+     image: ubuntu:24.04
+     variables:
+       GET_GCP_SECRET_MY_SECRET: projects/my-project/secrets/my-secret/versions/latest
+     before_script:
+       - apt-get update && apt-get install -y curl jq # Required as this example uses the ubuntu:24.04 base image.
+       - !reference [.get-gcp-secrets, before_script]
+     script: do-something --secret $MY_SECRET
+   ```
+
+## Specifying secrets
+
+The `before_script` looks for all variables available to the job which begin
+with `GET_GCP_SECRET_`. It expects that each of these variables contains the
+full id of the secret version to be retrieved e.g.
+`projects/my-project/secrets/my-secret/versions/latest`
+
+Each secret version is retrieved and exported to an environment variable named
+with the variable name minus the `GET_GCP_SECRET_` prefix. For example, if you
+use the variable name `GET_GCP_SECRET_API_TOKEN` the secret value will be
+exported to the `API_TOKEN` environment variable.
+
+Providing your runner has the appropriate permissions, it is possible to define
+multiple secrets in a single job. For example:
+
+```yaml
+  my-job:
+    stage: build
+    image: ubuntu:24.04
+    variables:
+      GET_GCP_SECRET_SUPER_SECRET_TOKEN: projects/some-project-1/secrets/some-secret-1/versions/latest
+      GET_GCP_SECRET_DATABASE_PASSWORD: projects/some-project-2/secrets/some-secret-2/versions/56
+    before_script:
+      - apt-get update && apt-get install -y curl jq # Required as this example uses the ubuntu:24.04 base image.
+      - !reference [.get-gcp-secrets, before_script]
+    script: |
+      do-something --secret $SUPER_SECRET_TOKEN
+      do-something-else --database-password $DATABASE_PASSWORD
+```
+
+## Google Service Account impersonation
+
+In some situations, the identity of your CI runner may not have permission to
+access the required secrets itself but it may be able to impersonate a service
+account which does. To specify the email address of a service account to be
+impersonated you can use the `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` variable.
+
+```yaml
+  my-job:
+    stage: build
+    image: ubuntu:24.04
+    variables:
+      GOOGLE_IMPERSONATE_SERVICE_ACCOUNT: my-service-account@project-a.iam.gserviceaccount.com
+      GET_GCP_SECRET_SUPER_SECRET_TOKEN: projects/some-project-1/secrets/some-secret-1/versions/latest
+      GET_GCP_SECRET_DATABASE_PASSWORD: projects/some-project-2/secrets/some-secret-2/versions/56
+    before_script:
+      - apt-get update && apt-get install -y curl jq # Required as this example uses the ubuntu:24.04 base image.
+      - !reference [.get-gcp-secrets, before_script]
+    script: |
+      do-something --secret $SUPER_SECRET_TOKEN
+      do-something-else --database-password $DATABASE_PASSWORD
+```

fragments/get-gcp-secrets.yml

+# This is a helper script to access GCP secrets in pipeline jobs. For more information see ./get-gcp-secrets.md.
+
+.get-gcp-secrets:
+  before_script:
+    - |
+      set -e
+      echo "Retrieving an access token for the default service account."
+      default_access_token_res=$(
+          curl --fail-with-body -s -S -H "Metadata-Flavor: Google" \
+              "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token"
+      )
+      access_token=$(echo "$default_access_token_res" | jq -r ".access_token")
+
+      if [[ -n "$GOOGLE_IMPERSONATE_SERVICE_ACCOUNT" ]]; then
+          echo "Impersonating the $GOOGLE_IMPERSONATE_SERVICE_ACCOUNT service account."
+          access_token_res=$(
+              curl --fail-with-body -s -S -X POST \
+              -H "Authorization: Bearer $access_token" \
+              -H "Content-Type: application/json; charset=utf-8" \
+              "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT/@/%40}:generateAccessToken" \
+              -d '{"scope": ["https://www.googleapis.com/auth/cloud-platform"]}'
+          )
+          access_token=$(echo "$access_token_res" | jq -r ".accessToken")
+      fi
+
+      # This loops through all variables found which begin with "GET_GCP_SECRET_". Each variable relates to a single
+      # Google Secret Manager secret, the value of which is retrieved and exported as a new variable. The exported
+      # variable name is the original variable name with the "GET_GCP_SECRET_" prefix removed. For example,
+      # "GET_GCP_SECRET_GITLAB_TOKEN" will be exported as "GITLAB_TOKEN".
+      for secret_var in "${!GET_GCP_SECRET_@}"; do
+          var_name_to_export="${secret_var/GET_GCP_SECRET_/}"
+          secret_var_value="${!secret_var}"
+          
+          echo "Retrieving secret value from $secret_var_value."
+          secret_res=$(
+              curl --fail-with-body -s -S -H "Authorization: Bearer $access_token" \
+                  -H "Content-Type: application/json" \
+                  "https://secretmanager.googleapis.com/v1/$secret_var_value:access"
+          )
+          secret_data=$(echo "$secret_res" | jq -r ".payload.data" | base64 -d)
+          echo "Exporting secret as $var_name_to_export."
+          export "$var_name_to_export=$secret_data"
+      done