Initial stacks POC
Introduction
The Cloud team are looking for a tool that provides functionality similar to HashiCorp Terraform Stacks but better aligns with our specific needs. After evaluating currently available solutions (mainly HCP Terraform Stacks and Terragrunt), we have found that none fully meet our requirements. The ideal tool must be flexible enough to support multiple use cases, including separating our Kubernetes cluster deployments from the workloads deployed into the cluster, and provisioning our multiple cloud platform factories in a single "stack". This proof of concept (POC) explores a potential in-house solution to address these needs and assesses its feasibility within our environment.
Concept
The proposed tool aims to provide a lightweight automation layer around the deployment of one or more Terraform root modules. Rather than introducing complex automation or "magic", it will act as a simple orchestrator, ensuring that Terraform root modules are deployed in a structured and efficient manner.
Conceptually, this tool can be thought of as Docker Compose for Terraform root modules — a declarative way to define and coordinate the deployment of multiple Terraform configurations.
Key requirements
The proposed tool must meet the following key requirements as part of this POC:
- Must be written in Python, using our standard toolchain and CI pipelines.
- Terraform root modules (components) available in the stack should be defined in a separate "components"
yamlfile.- The source for these components can be either local directories or remote git repositories.
- For git repositories we should be able to specify version constraints and the source code should be cloned to a
.ucam-stacksdirectory for the stack.
- The deployment(s) of the available components in a stack, and any required input variables etc., should be defined in a separate "deployments"
yamlfile. - Must support our two key use cases:
- Splitting Kubernetes deployments into two separate Terraform stages
- This is a key use case for HashiCorp's Stacks offering and has long been an issue in the Terraform community.
- Merging one or more of our cloud platform factories into a single "meta-factory" stack.
- Splitting Kubernetes deployments into two separate Terraform stages
- Root modules should be able to depend on the outputs of other root modules.
- This requires creating and managing a dependency graph for all deployments in the stack, which thanks to the graphlib library is pretty trivial
😄
- This requires creating and managing a dependency graph for all deployments in the stack, which thanks to the graphlib library is pretty trivial
- Root modules that are ready to be deployed (i.e., not waiting for dependencies) should be executed in parallel.
- The dependency graph should be consumable as a Python library, allowing other tooling to integrate with it — for example, enabling the construction of dynamic CI pipelines based on the graph's structure.