FAQ | This is a LIVE service | Changelog

Skip to content
Commits on Source (6)
Hide whitespace changes
Inline Side-by-side
CHANGELOG.md
View file @ 185bcaf3
......@@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [4.1.0] - 2024-05-16
### Added
- A generic `get-gcp-secrets` fragment to retrieve one or more Google Secret
Manager secrets in a CI job.
## [4.0.0] - 2024-04-05
### Changed
......
auto-devops/release-it.yml
View file @ 185bcaf3
# This template enables automated release management using the release-it tool.
# This template enables automated release management using the release-it tool.
include:
- local: /fragments/get-gcp-secrets.yml
.release-base:
image:
......@@ -6,50 +9,34 @@
variables:
GIT_STRATEGY: clone
GIT_DEPTH: 0
before_script: |
MAGENTA="\e[35m"
CLEAR="\e[0m"
log() {
echo -e "${MAGENTA}${1}${CLEAR}"
}
# By default, GitLab performs a shallow clone of the target repository and checks out a detached commit rather than
# the branch itself. We want to be able to search the full history so we need to do the following.
git fetch origin
git checkout "$CI_COMMIT_REF_NAME"
log "Retrieving an access token for the default service account of the runner pod..."
DEFAULT_TOKEN=$(
curl --fail-with-body -s -S -H "Metadata-Flavor: Google" \
"http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
| jq -r ".access_token"
)
log "Generating an access token for the $GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT service account..."
ACCESS_TOKEN=$(
curl --fail-with-body -s -S -X POST \
-H "Authorization: Bearer $DEFAULT_TOKEN" \
-H "Content-Type: application/json; charset=utf-8" \
"https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT/@/%40}:generateAccessToken" \
-d '{"scope": ["https://www.googleapis.com/auth/cloud-platform"]}' \
| jq -r ".accessToken"
)
log "Retrieving the GitLab bot access token from Google Secret Manager..."
export GITLAB_TOKEN=$(
curl --fail-with-body -s -S -H "Authorization: Bearer $ACCESS_TOKEN" \
-H "Content-Type: application/json" \
"https://secretmanager.googleapis.com/v1/$GITLAB_ACCESS_TOKEN_SECRET_ID/versions/latest:access" \
| jq -r ".payload.data" | base64 -d
)
before_script:
# If our standard GITLAB_ACCESS_TOKEN_SECRET_ID variable is set we re-export it to a variable with the correct
# name/value for the get-gcp-secrets helper script.
- |
if [[ -n "$GITLAB_ACCESS_TOKEN_SECRET_ID" ]]; then
export GET_GCP_SECRET_GITLAB_TOKEN="$GITLAB_ACCESS_TOKEN_SECRET_ID/versions/latest"
fi
# If our standard GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT variable is set we re-export it as the generic
# GOOGLE_IMPERSONATE_SERVICE_ACCOUNT variable that the get-gcp-secrets helper script expects.
- |
if [[ -n "$GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT" ]]; then
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT="$GITLAB_TOKEN_ACCESSOR_SERVICE_ACCOUNT"
fi
# Call the get-gcp-secrets helper script to retrieve the $GET_GCP_SECRET_GITLAB_TOKEN secret value and export it as
# GITLAB_TOKEN for the release-it tool to access.
- !reference [.get-gcp-secrets, before_script]
- |
# By default, GitLab performs a shallow clone of the target repository and checks out a detached commit rather than
# the branch itself. We want to be able to search the full history so we need to do the following.
git fetch origin
git checkout "$CI_COMMIT_REF_NAME"
log "Configuring git to use the GitLab bot account..."
git config user.email "$GITLAB_ACCESS_TOKEN_EMAIL"
git config user.name "$GITLAB_ACCESS_TOKEN_NAME"
echo "Configuring git to use the GitLab bot account..."
git config user.email "${GITLAB_ACCESS_TOKEN_EMAIL:-release-it-bot@noreply.gitlab.developers.cam.ac.uk}"
git config user.name "${GITLAB_ACCESS_TOKEN_NAME:-release-it-bot}"
log "Configuring the git remote url..."
git remote set-url origin "https://token:$GITLAB_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
echo "Configuring the git remote url..."
git remote set-url origin "https://token:$GITLAB_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
tags:
- $GKE_RUNNER_TAG
......
fragments/get-gcp-secrets.md 0 → 100644
View file @ 185bcaf3
# GCP Secrets Helper
Accessing secrets securely in CI/CD pipelines is notoriously tricky. This
fragment provides some reusable code to make the process of retrieving one
or more Google Secret Manager secrets easier.
## Prerequisites
To use this fragment in your CI jobs you will need to ensure that the following
prerequisites are met.
- You'll need to ensure that `bash`, `curl`, and `jq` are installed in your
jobs' image and that your job is running in a `bash` shell.
- The CI runner executing your job must be running in the context of a Google
Cloud identity. The identity must either have the `secrets.secretAccessor`
role granted for all secrets to be retrieved, or have the ability to
impersonate a service account which does.
## How-to
The following steps will retrieve the `latest` version of the Google Secret
Manager secret named `my-secret` from the Google Cloud project named
`my-project` and export the secret value as an environment variable named
`MY_SECRET`.
1. First, in your `.gitlab-ci.yml` file include the `get-gcp-secrets.yml`
template.
```yaml
include:
- project: uis/devops/continuous-delivery/ci-templates
file: /fragments/get-gcp-secrets.yml
ref: <specify version here>
```
1. Next, define a job with the variable `GET_GCP_SECRET_MY_SECRET:
projects/my-project/secrets/my-secret/versions/latest`.
```yaml
my-job:
stage: build
image: ubuntu:24.04
variables:
GET_GCP_SECRET_MY_SECRET: projects/my-project/secrets/my-secret/versions/latest
```
1. Specify the `$GKE_RUNNER_TAG` job tag. When working on UIS DevOps projects,
you'll usually need to run your CI jobs on one of our Google Kubernetes
Engine hosted runners to be able to use this helper. To do that, add the
magic `$GKE_RUNNER_TAG` variable to the list of tags for your job.
```yaml
my-job:
stage: build
image: ubuntu:24.04
variables:
GET_GCP_SECRET_MY_SECRET: projects/my-project/secrets/my-secret/versions/latest
tags:
- $GKE_RUNNER_TAG
```
1. Add a reference to the included hidden `.get-gcp-secrets` job's
`before_script` block in your job's `before_script`.
```yaml
my-job:
stage: build
image: ubuntu:24.04
variables:
GET_GCP_SECRET_MY_SECRET: projects/my-project/secrets/my-secret/versions/latest
before_script:
- apt-get update && apt-get install -y curl jq # Required as this example uses the ubuntu:24.04 base image.
- !reference [.get-gcp-secrets, before_script]
```
1. The `before_script` referenced in `.get-gcp-secrets` will retrieve the value
of the secret in the `projects/my-project/secrets/my-secret/versions/latest`
secret object and export it as an environment variable named `MY_SECRET`.
This variable can then be referenced in the `script` block of your job in the
usual way.
```yaml
my-job:
stage: build
image: ubuntu:24.04
variables:
GET_GCP_SECRET_MY_SECRET: projects/my-project/secrets/my-secret/versions/latest
before_script:
- apt-get update && apt-get install -y curl jq # Required as this example uses the ubuntu:24.04 base image.
- !reference [.get-gcp-secrets, before_script]
script: do-something --secret $MY_SECRET
```
## Specifying secrets
The `before_script` looks for all variables available to the job which begin
with `GET_GCP_SECRET_`. It expects that each of these variables contains the
full id of the secret version to be retrieved e.g.
`projects/my-project/secrets/my-secret/versions/latest`
Each secret version is retrieved and exported to an environment variable named
with the variable name minus the `GET_GCP_SECRET_` prefix. For example, if you
use the variable name `GET_GCP_SECRET_API_TOKEN` the secret value will be
exported to the `API_TOKEN` environment variable.
Providing your runner has the appropriate permissions, it is possible to define
multiple secrets in a single job. For example:
```yaml
my-job:
stage: build
image: ubuntu:24.04
variables:
GET_GCP_SECRET_SUPER_SECRET_TOKEN: projects/some-project-1/secrets/some-secret-1/versions/latest
GET_GCP_SECRET_DATABASE_PASSWORD: projects/some-project-2/secrets/some-secret-2/versions/56
before_script:
- apt-get update && apt-get install -y curl jq # Required as this example uses the ubuntu:24.04 base image.
- !reference [.get-gcp-secrets, before_script]
script: |
do-something --secret $SUPER_SECRET_TOKEN
do-something-else --database-password $DATABASE_PASSWORD
```
## Google Service Account impersonation
In some situations, the identity of your CI runner may not have permission to
access the required secrets itself but it may be able to impersonate a service
account which does. To specify the email address of a service account to be
impersonated you can use the `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` variable.
```yaml
my-job:
stage: build
image: ubuntu:24.04
variables:
GOOGLE_IMPERSONATE_SERVICE_ACCOUNT: my-service-account@project-a.iam.gserviceaccount.com
GET_GCP_SECRET_SUPER_SECRET_TOKEN: projects/some-project-1/secrets/some-secret-1/versions/latest
GET_GCP_SECRET_DATABASE_PASSWORD: projects/some-project-2/secrets/some-secret-2/versions/56
before_script:
- apt-get update && apt-get install -y curl jq # Required as this example uses the ubuntu:24.04 base image.
- !reference [.get-gcp-secrets, before_script]
script: |
do-something --secret $SUPER_SECRET_TOKEN
do-something-else --database-password $DATABASE_PASSWORD
```
fragments/get-gcp-secrets.yml 0 → 100644
View file @ 185bcaf3
# This is a helper script to access GCP secrets in pipeline jobs. For more information see ./get-gcp-secrets.md.
.get-gcp-secrets:
before_script:
- |
set -e
echo "Retrieving an access token for the default service account."
default_access_token_res=$(
curl --fail-with-body -s -S -H "Metadata-Flavor: Google" \
"http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token"
)
access_token=$(echo "$default_access_token_res" | jq -r ".access_token")
if [[ -n "$GOOGLE_IMPERSONATE_SERVICE_ACCOUNT" ]]; then
echo "Impersonating the $GOOGLE_IMPERSONATE_SERVICE_ACCOUNT service account."
access_token_res=$(
curl --fail-with-body -s -S -X POST \
-H "Authorization: Bearer $access_token" \
-H "Content-Type: application/json; charset=utf-8" \
"https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT/@/%40}:generateAccessToken" \
-d '{"scope": ["https://www.googleapis.com/auth/cloud-platform"]}'
)
access_token=$(echo "$access_token_res" | jq -r ".accessToken")
fi
# This loops through all variables found which begin with "GET_GCP_SECRET_". Each variable relates to a single
# Google Secret Manager secret, the value of which is retrieved and exported as a new variable. The exported
# variable name is the original variable name with the "GET_GCP_SECRET_" prefix removed. For example,
# "GET_GCP_SECRET_GITLAB_TOKEN" will be exported as "GITLAB_TOKEN".
for secret_var in "${!GET_GCP_SECRET_@}"; do
var_name_to_export="${secret_var/GET_GCP_SECRET_/}"
secret_var_value="${!secret_var}"
echo "Retrieving secret value from $secret_var_value."
secret_res=$(
curl --fail-with-body -s -S -H "Authorization: Bearer $access_token" \
-H "Content-Type: application/json" \
"https://secretmanager.googleapis.com/v1/$secret_var_value:access"
)
secret_data=$(echo "$secret_res" | jq -r ".payload.data" | base64 -d)
echo "Exporting secret as $var_name_to_export."
export "$var_name_to_export=$secret_data"
done
terraform-lint.yml
View file @ 185bcaf3
......@@ -83,6 +83,65 @@ tflint:
# results which is viewable in the merge request UI. To ignore specific checks see -
# https://aquasecurity.github.io/trivy/v0.50/docs/configuration/filtering/#trivyignoreyaml. By default, we're ignoring
# certain checks which we have decided to allow due to our boilerplate template design.
# Excluded general checks:
# - name: AVD-GCP-0017
# link: https://avd.aquasec.com/misconfig/google/sql/avd-gcp-0017/
# description: Ensure that Cloud SQL Database Instances are not publicly exposed
# severity: High
# reason: It is recommended that database instances should be configured so that they are not available over
# the public internet, but to internal compute resources that access them. Valid recommendation, but not
# always applicable due to our system design limitations.
# - name: AVD-GCP-0029
# link: https://avd.aquasec.com/misconfig/google/compute/avd-gcp-0029/
# description: VPC flow logs should be enabled for all subnetworks
# severity: Low
# reason: Just a recommendation to have this feature enabled as VPC flow logs record
# information about all traffic, which can help with reviewing anomalous traffic.
# - name: AVD-GCP-0066
# link: https://avd.aquasec.com/misconfig/google/storage/avd-gcp-0066/
# description: Cloud Storage buckets should be encrypted with a customer-managed key
# severity: Low
# reason: It is recommended to use customer-managed keys. Using unmanaged keys makes rotation and general management
# difficult. However, in our case, it might bring additional operational overhead so this check is disabled.
# Excluded PostgreSQL configuration-related checks:
# - name: AVD-GCP-0014
# link: https://avd.aquasec.com/misconfig/google/sql/avd-gcp-0014/
# description: Temporary file logging should be enabled for all temporary files
# severity: Medium
# reason: Trivy suggests setting `settings.database_flags["log_temp_files"]` to "0" for sql instance. Default value in
# PostgreSQL is "-1" (i.e. disabled). See https://pgpedia.info/l/log_temp_files.html for more information.
# - name: AVD-GCP-0016
# link: https://avd.aquasec.com/misconfig/google/sql/avd-gcp-0016/
# description: Ensure that logging of connections is enabled
# severity: Medium
# reason: Trivy suggests setting `settings.database_flags["log_connections"]` to "on" for sql instance.
# Default value in PostgreSQL is "off" (i.e. disabled). By default, PostgreSQL only logs the errors
# generated by unsuccessful connections. Setting this option to "on" will log all.
# See https://pgpedia.info/l/log_connections.html for more information.
# - name: AVD-GCP-0020
# link: https://avd.aquasec.com/misconfig/google/sql/avd-gcp-0020/
# description: Ensure that logging of lock waits is enabled.
# severity: Medium
# reason: Trivy suggests setting `settings.database_flags["log_lock_waits"]` to "on" for sql instance.
# Default value in PostgreSQL is "off" (i.e. disabled). A configuration parameter determining whether a log
# message is produced when a session waits longer than deadlock_timeout to acquire a lock.
# See https://pgpedia.info/l/log_lock_waits.html for more information.
# - name: AVD-GCP-0022
# link: https://avd.aquasec.com/misconfig/google/sql/avd-gcp-0022/
# description: Ensure that logging of disconnections is enabled.
# severity: Medium
# reason: Trivy suggests setting `settings.database_flags["log_disconnections"]` to "on" for sql instance.
# Default value in PostgreSQL is "off" (i.e. disabled). Logging disconnections provides useful diagnostic
# data such as session length, which can identify performance issues in an application and
# potential DoS vectors. See https://pgpedia.info/l/log_connections.html for more information.
# - name: AVD-GCP-0025
# link: https://avd.aquasec.com/misconfig/google/sql/avd-gcp-0025/
# description: Ensure that logging of checkpoints is enabled
# severity: Medium
# reason: Trivy suggests setting `settings.database_flags["log_checkpoints"]` to "on" for sql instance.
# This is disabled in PostgreSQL 8.3 ~ PostgreSQL 14 and enabled by default since PostgreSQL 15.
# Different projects use different versions so this check is disabled.
# See https://pgpedia.info/l/log_checkpoints.html for more information.
trivy:
stage: test
image:
......@@ -90,14 +149,14 @@ trivy:
entrypoint: [""]
variables:
TRIVY_IGNORE: |-
AVD-GCP-0066
AVD-GCP-0029
AVD-GCP-0014
AVD-GCP-0017
AVD-GCP-0025
AVD-GCP-0016
AVD-GCP-0022
AVD-GCP-0017
AVD-GCP-0020
AVD-GCP-0022
AVD-GCP-0025
AVD-GCP-0029
AVD-GCP-0066
before_script: |
mkdir ${TF_DATA_DIR}
for i in $TRIVY_IGNORE; do echo $i >> .trivyignore; done
......