initial implementation

Copied directly from
https://gitlab.developers.cam.ac.uk/uis/devops/api/apigee-ops

.editorconfig

+root=true
+
+[*.tf]
+indent_style=space
+indent_size=2

.gitlab-ci.yml

+# The "test" job will run the following tests:
+#
+# Terraform's "fmt" utility does not request any changes (i.e. a code lint)
+test:
+  image: docker:stable-git
+
+  services:
+    - docker:stable-dind
+
+  script:
+    # Run terraform's format checker.
+    - docker run -v $(pwd):/deploy --rm hashicorp/terraform:light fmt -check -diff /deploy

CHANGELOG

+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2021-03-16
+### Added
+ - Initial version

LICENSE.txt

+Copyright (c) 2021, University of Cambridge Information Services
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

-# GCP Scheduled Script
+# Scheduled script run within Google Cloud
 
-Run a Python script regularly using a cronjob-style schedule
\ No newline at end of file
+This terraform module configures a single-file Python script to be run regularly
+in Google Cloud using a cron-style scheduling syntax. It supports single-file
+scripts with requirements specified via a `requirements.txt`-style list of
+packages.
+
+The scripts may have secrets passed to them by a Google Secret Manager secret
+created for this script.
+
+A dedicated service account is created for the script which represents the
+identity which the script runs as within Google Cloud.
+
+Monitoring is configured with alerts sent if the number of successful
+invocations of the script within a configurable time period falls below a given
+threshold.
+
+This module is not suitable for multi-file scripts or scripts which need to be
+passed parameters which differ from run-to-run.
+
+## Versioning
+
+The `master` branch contains the tip of development and corresponds to the `v1`
+branch. Each release is tagged with its bare version number.
+
+## Required APIs and services
+
+The following APIs must be enabled to use this module:
+
+* `appengine.googleapis.com`
+* `cloudbuild.googleapis.com`
+* `cloudfunctions.googleapis.com`
+* `cloudscheduler.googleapis.com`
+* `secretmanager.googleapis.com`
+
+In addition there must be an app engine application configured for your project.
+This is a requirement to configure Cloud Scheduler jobs. You can ensure an app
+engine application is configured by means of the `google_app_engine_application`
+resource:
+
+```tf
+resource "google_app_engine_application" "app" {
+  location_id = "europe-west2"
+}
+```
+
+## Monitoring
+
+Cloud Monitoring resources are associated with a Monitoring Workspace which may
+be hosted in a project which differs from the project containing other
+resources. The `google.monitoring` provider is used when creating Monitoring
+resources and it should be a provider configured with credentials able to create
+monitoring resources.
+
+## Implementation
+
+The script is implemented as a Cloud Function and so it should conform to the
+[Cloud Function Python
+runtime](https://cloud.google.com/functions/docs/concepts/python-runtime). The
+entrypoint to the function can be customised but the default is to use `main`.
+
+A dedicated Cloud Storage bucket is created to store the source code in. This is
+created with a random name.
+
+## Example
+
+A minimal example of running a script:
+
+```tf
+module "scheduledscript" {
+  source = "..."
+
+  # Name used to form resource names and human readable description.
+  name        = "my-script"
+  description = "Log a message."
+
+  # Project and region to create resources in.
+  project = "..."
+  region  = "europe-west2"
+
+  # A directory which can be used to store local files which are not checked in
+  # to source control.
+  local_files_dir = "/terraform_data/local"
+
+  # The script itself.
+  script = <<-EOL
+      import logging
+
+      LOG = logging.getLogger()
+      logging.basicConfig(level=logging.INFO)
+
+      def main(request):
+          LOG.info('I ran!')
+          return '{ "status": "ok" }', {'Content-Type': 'application/json'}
+  EOL
+
+  # Cron-style schedule for running the job.
+  schedule = "*/5 * * * *"
+
+  # Alerting threshold. Specify the minimum number of successful invocations and
+  # the period over which this should be measured.
+  alert_success_threshold = 5
+  alert_success_period    = "3600s"  # == 1hr
+
+  # Email addresses to receive alerts if invocations are failing.
+  alert_email_addresses = ["someone@example.com"]
+
+  # Use a separate provider with rights over the Cloud Monitoring workspace for
+  # monitoring resource management.
+  providers = {
+    google.monitoring = google.monitoring
+  }
+}
+```
+
+See the [variables.tf](variables.tf) file for all variables.
+
+Additional Python package requirements can be specified in the `requirements`
+variable which should be a list of Python packages, one per line, as used by
+`requirements.txt` files.
+
+A custom entry point for the script can be configured via the `entry_point`
+variable.
+
+Configurations can be passed to the application via the `secret_configuration`
+variable. This should be a string which is placed in a Google Secret Manager
+secret and is made available to the script. The secret is passed as a URL of the
+form `sm://[PROJECT]/[SECRET]#[VERSION]` in the `SECRET_CONFIGURATION_SOURCE`
+environment variable.
+
+## Outputs
+
+See the [outputs.tf](outputs.tf) file for all outputs.
+
+The Cloud Function resource created to run the script is available via the
+`function` output.
+
+The Service Account identity the script is run as is available via the
+`service_account` output.
+
+The Scheduler job which is configured to run the script is available via the
+`job` output.

main.tf

+# main.tf defines resources for the module
+
+# The google_project data source is used to get the default projects for the
+# Google providers.
+data "google_project" "provider_project" {}
+
+data "google_project" "monitoring_project" {
+  provider = google.monitoring
+}
+
+locals {
+  # Allow the project and monitoring_project variables to override the provider
+  # projects.
+  project            = coalesce(var.project, data.google_project.provider_project.project_id)
+  monitoring_project = coalesce(var.monitoring_project, data.google_project.monitoring_project.project_id)
+
+  # A kebab-case name in case we're given underscore_case.
+  kebab_name = replace(var.name, "_", "-")
+}
+
+# The source code of the script.
+data "archive_file" "source" {
+  type = "zip"
+
+  output_path = "${var.local_files_dir}/scripts/${var.name}-source.zip"
+
+  source {
+    content  = var.requirements
+    filename = "requirements.txt"
+  }
+
+  source {
+    content  = var.script
+    filename = "main.py"
+  }
+}
+
+# The source code lives as an object in a dedicated bucket.
+resource "random_id" "source_bucket_name" {
+  byte_length = 4
+  prefix      = "${local.kebab_name}-source-"
+}
+
+resource "google_storage_bucket" "source" {
+  project  = local.project
+  name     = random_id.source_bucket_name.hex
+  location = var.region
+}
+
+resource "google_storage_bucket_object" "source" {
+  name   = "source-${data.archive_file.source.output_base64sha256}.zip"
+  bucket = google_storage_bucket.source.name
+  source = data.archive_file.source.output_path
+}
+
+# A service account which is used to run the script within a Cloud Function.
+resource "google_service_account" "script" {
+  project      = local.project
+  account_id   = "script-${local.kebab_name}"
+  display_name = "${var.name} script identity"
+}
+
+# A Secret Manager secret which holds configuration for the script.
+module "secret_configuration" {
+  source = "git::https://gitlab.developers.cam.ac.uk/uis/devops/infra/terraform/gcp-secret-manager.git"
+
+  project     = local.project
+  region      = var.region
+  secret_id   = "${local.kebab_name}-configuration"
+  secret_data = var.secret_configuration
+}
+
+# The script itself as a Cloud Function.
+resource "google_cloudfunctions_function" "script" {
+  project = local.project
+  region  = var.region
+
+  name        = var.name
+  description = var.description
+  runtime     = var.runtime
+
+  timeout     = var.timeout
+  entry_point = var.entry_point
+
+  service_account_email = google_service_account.script.email
+
+  source_archive_bucket = google_storage_bucket.source.name
+  source_archive_object = google_storage_bucket_object.source.name
+
+  trigger_http        = true
+  available_memory_mb = var.available_memory_mb
+  max_instances       = 1
+
+  environment_variables = {
+    SECRET_CONFIGURATION_SOURCE = module.secret_configuration.url
+  }
+}
+
+# Allow the service account to invoke the function.
+resource "google_cloudfunctions_function_iam_member" "function_invoker" {
+  project        = google_cloudfunctions_function.script.project
+  region         = google_cloudfunctions_function.script.region
+  cloud_function = google_cloudfunctions_function.script.name
+
+  role   = "roles/cloudfunctions.invoker"
+  member = "serviceAccount:${google_service_account.script.email}"
+}
+
+# Allow the service account to read the configuration secret.
+resource "google_secret_manager_secret_iam_member" "secret_reader" {
+  project   = module.secret_configuration.project
+  secret_id = module.secret_configuration.secret_id
+
+  role   = "roles/secretmanager.secretAccessor"
+  member = "serviceAccount:${google_service_account.script.email}"
+}
+
+# A scheduled job to run the script
+resource "google_cloud_scheduler_job" "script" {
+  project = local.project
+  region  = var.region
+
+  name             = var.name
+  description      = var.description
+  schedule         = var.schedule
+  time_zone        = var.time_zone
+  attempt_deadline = "${var.timeout}s"
+
+  retry_config {
+    retry_count = 1
+  }
+
+  http_target {
+    http_method = "POST"
+    uri         = google_cloudfunctions_function.script.https_trigger_url
+    body        = base64encode("{}")
+
+    oidc_token {
+      service_account_email = google_service_account.script.email
+      audience = (
+        google_cloudfunctions_function.script.https_trigger_url
+      )
+    }
+  }
+}
+
+# Email notification channels for alerts.
+resource "google_monitoring_notification_channel" "email" {
+  for_each = toset(var.alert_email_addresses)
+
+  project = local.monitoring_project
+
+  display_name = each.value
+  type         = "email"
+
+  labels = {
+    email_address = each.value
+  }
+
+  provider = google.monitoring
+}
+
+# A custom log metric used to select successful invocation of the script via the
+# Cloud Scheduler job. Logging metrics can live in the project hosting logs
+# since, counter-intuitively, it is not a Cloud Monitoring resource.
+resource "google_logging_metric" "successes" {
+  project = local.project
+
+  name = "script/successes-${local.kebab_name}"
+
+  filter = trimspace(replace(
+    <<-EOI
+      resource.type="cloud_scheduler_job"
+      AND resource.labels.project_id = "${local.project}"
+      AND resource.labels.job_id = "${google_cloud_scheduler_job.script.name}"
+      AND httpRequest.status=200
+    EOI
+  , "\n", " "))
+
+  metric_descriptor {
+    metric_kind = "DELTA"
+    value_type  = "INT64"
+  }
+}
+
+# Alerting if the time between successful POST-s to the script endpoint is
+# greater than a threshold.
+resource "google_monitoring_alert_policy" "successes" {
+  project = local.monitoring_project
+
+  display_name = "Invocations of ${var.name} script succeeds (${terraform.workspace})"
+
+  enabled = var.alert_enabled
+
+  notification_channels = [
+    for address in var.alert_email_addresses :
+    google_monitoring_notification_channel.email[address].name
+  ]
+
+  combiner = "OR"
+
+  conditions {
+    # Include the workspace name in the alert text to aid triaging alerts.
+    display_name = "Invocations of ${var.name} script succeeds (${terraform.workspace})"
+
+    condition_threshold {
+      # We're interested in Cloud Scheduler successes for the job in the
+      # appropriate project.
+      filter = trimspace(replace(
+        <<-EOI
+          metric.type="logging.googleapis.com/user/${google_logging_metric.successes.id}"
+          AND resource.type="cloud_scheduler_job"
+          AND resource.label.project_id="${local.project}"
+        EOI
+        , "\n", " "
+      ))
+
+      # We check our threshold value every minute.
+      duration   = "60s"
+      comparison = "COMPARISON_LT"
+
+      # The metric value is the number of successes in a given period.
+      aggregations {
+        alignment_period     = var.alert_success_period
+        per_series_aligner   = "ALIGN_SUM"
+        cross_series_reducer = "REDUCE_SUM"
+      }
+
+      threshold_value = var.alert_success_threshold
+    }
+  }
+
+  provider = google.monitoring
+}

outputs.tf

+# outputs.tf describes the outputs from the module
+
+output "function" {
+  value = google_cloudfunctions_function.script
+
+  description = <<-EOT
+    google_cloudfunctions_function resource for the Cloud Function created to
+    run the script.
+  EOT
+}
+
+output "service_account" {
+  value = google_service_account.script
+
+  description = <<-EOT
+    google_service_account resource for the Service Account which the script
+    runs as.
+  EOT
+}
+
+output "job" {
+  value = google_cloud_scheduler_job.script
+
+  description = <<-EOT
+    google_cloud_scheduler_job resource for the Scheduled Job which runs the script.
+  EOT
+}

providers.tf

+# providers.tf configures providers for the module.
+
+# A dedicated provider which is used to configure Cloud Monitoring resources.
+provider "google" {
+  alias = "monitoring"
+}

variables.tf

+# variables.tf defines variables used by the module
+
+variable "name" {
+  description = "Short resource-friendly name for the script."
+}
+
+variable "local_files_dir" {
+  description = <<-EOT
+    A local directory where files may be created which persist between runs but
+    which are not checked into source control.
+  EOT
+}
+
+variable "description" {
+  default     = ""
+  description = "Longer human-friendly description of the script"
+}
+
+variable "project" {
+  default     = ""
+  description = "Project to create resources in. Defaults to provider project."
+}
+
+variable "monitoring_project" {
+  default     = ""
+  description = <<-EOT
+    Project to create Cloud Monitoring resources in. Defaults to the project
+    used by the google.monitoring provider if not specified.
+  EOT
+}
+
+variable "region" {
+  default     = "europe-west2"
+  description = "Region to create resources in. Defaults to London, UK."
+}
+
+variable "script" {
+  default     = <<-EOT
+    def main(request):
+        return 'ok'
+  EOT
+  description = <<-EOT
+    The script to run. This should match the requirements for the Cloud Function
+    Python runtime. The entrypoint defaults to "main" unless overridden via the
+    "entry_point" variable.
+  EOT
+}
+
+variable "entry_point" {
+  default     = "main"
+  description = "Entrypoint to script. Defaults to 'main'."
+}
+
+variable "requirements" {
+  default     = ""
+  description = "requirements.txt-style file containing script dependencies"
+}
+
+variable "schedule" {
+  default     = "*/30 * * * *"
+  description = <<-EOT
+    Schedule for script running. Defaults to twice per hour. See
+    https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules for a
+    description of the format.
+  EOT
+}
+
+variable "time_zone" {
+  default     = "Europe/London"
+  description = <<-EOT
+    Time zone which "schedule" should be evaluated in. Default: 'Europe/London'.
+  EOT
+}
+
+variable "timeout" {
+  default     = 120
+  description = <<-EOT
+    Maximum time, in seconds, that a script can take to execute. Invocations
+    which take longer than this fail. Default: 120 seconds.
+  EOT
+}
+
+variable "available_memory_mb" {
+  default     = 128
+  description = <<-EOT
+    Maxiumum memory available to the script in MiB. Default: 128 MiB.
+  EOT
+}
+
+variable "alert_email_addresses" {
+  type        = list(string)
+  default     = []
+  description = <<-EOT
+    List of email addresses which should be notified if invocations of the
+    script start to fail.
+  EOT
+}
+
+variable "alert_success_threshold" {
+  default     = 5
+  description = <<-EOT
+    The minimum number of successes within 'alert_success_period' below which an
+    alert is fired.
+  EOT
+}
+
+variable "alert_success_period" {
+  default     = "3600s"
+  description = <<-EOT
+    Period over which 'alert_success_threshold' is used. Default: "3600s" which
+    is one hour.
+  EOT
+}
+
+variable "alert_enabled" {
+  default     = true
+  description = <<-EOT
+    Flag indicating if alerting should be enabled for this script.
+  EOT
+}
+
+variable "secret_configuration" {
+  default     = ""
+  description = <<-EOT
+    Configuration which is placed in a Google Secret which can be read by the
+    service account identity which the script runs as. A path to the secret of
+    the form "sm://[PROJECT]/[SECRET]#[VERSION]" appears as the
+    "SECRET_CONFIGURATION_SOURCE" environment variable.
+  EOT
+}
+
+variable "runtime" {
+  default     = "python38"
+  description = <<-EOT
+    Python runtime for script. See
+    https://cloud.google.com/functions/docs/concepts/exec. Default: "python38".
+  EOT
+}

versions.tf

+# versions.tf defines minimum provider versions for the module
+
+terraform {
+  # Minimum provider versions.
+  required_providers {
+    google      = "~> 3.60"
+    google-beta = "~> 3.60"
+  }
+
+  # Minimum terraform version.
+  required_version = "~> 0.13"
+}