Explain our approach to secrets management

This issue captures discssions from https://gitlab.developers.cam.ac.uk/uis/devops/digital-admissions/albatross/albatross-deployment/-/issues/18#note_585200 along with private discussions with @rh841 around secrets management.

Although we have some reference documentation on secret-based configuration foir Django libraries we lack a more generic explainer which is language and framework neutral. As noted in uis/devops/tools/logan#28 (comment 586320), the reference documentation itself would benefit from using our gcp-secret-manager module.

The issue is to create an explainer which covers the following topics:

• What are "secrets" and what is "configuration"
• The different types of secret:
  • External (1password)
  • Generated (terraform)
• Ways of passing secrets:
  • Deployed services
  • Local development
  • GitLab CI jobs
• Passing external secrets in deployments
  • sanctuary exists and is intended to be a poor-man's data "1password_secret" resource.
  • Outline down-sides of using sanctuary to manage secrets mounted directly in applications:
    • Unclear when applications will re-read them (launch time versus every use is up to applications)
    • If one uses "latest", applications do not re-launch
    • If one specifies version, applications re-launch but may break since sanctuary deletes older versions.
• Passing internal secrets in deplotments
  • Not the same issues as sanctuary: should always pass with explicit version number
• Having a single canonical place secrets are stored:
  • For manually maintained secrets, this is 1password copied into GCP
  • For generated secrets, this is GCP
  • Prefer generated secrets over copied ones.
• Outline and justify a preferred approach:
  • Downsides of environment variables: one secret per environment variable, "leakiness"
  • Prefer mounting secrets as files. Allow configuration pointing to them.
  • Environment variables are OK if technically required.
• History of our current Django-based approach
• Recommendations for non-Django applications
• Include case studies of Spring Boot, Django and FastAPI-based services.