FAQ | This is a LIVE service | Changelog

Gitlab CI and secure environment variables

This is more of a use case possibility discussion and looking for ideas rather than a feature request.

As our team has been working through all the use cases here on Gitlab it's been working great so far. We've had no trouble with Windows .NET framework, .NET core, python and various ruby build environments. So far it's been great.

I'm starting to explore if we can deploy web applications to our internal Secure Data Hosting Service (SDHS) for medical data which is ISO 27001 certified. As part of this there are extremely strict controls and documentation on user access, logging, credentials, etc.

As part of a deploy setup we'd need to store some form of secrets on Gitlab to be able to allow it to deploy. The issue I'm looking at is how to guarantee restricted access which might not be possible without extra tooling. I've looked into the storage of these secrets within Gitlab and overall it seems secure but with an implicit trust of the administrators of the service.

The environment variables are encrypted at rest but seem to be unlocked via a master key through the settings. I think this is the chain:

Database schema - https://gitlab.com/gitlab-org/gitlab-ce/blob/master/db/schema.rb#L597 Crypto helper function - https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/crypto_helper.rb Settings key - https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/settings.rb#L98, https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/settings.rb#L110

The scenario I'm looking for is how to we store some credential in Gitlab CI that prevents a rogue administrator from extracting credentials from the Gitlab database configuration. I'll admit that this is extremely unlikely but to have not thought about it and mitigated the risk would be a big negligent!

Is there a way to guarantee complete control of secure credentials while still using the Gitlab CI environment?

Here's a couple of the scenarios I'm thinking of:

  1. Just put regular deploy credentials in Gitlab CI environment variables - The risk is these can be extracted from the database by a nefarious person even though it's extremely unlikely, sloppy .gitlab-ci.yml writing could leak them in the logs, etc. This is probably not acceptable to an ISO audit.
  2. Use of a tool like Hashicorp Vault - Limits access to a securable set of credentials but still risks a similar problem in that access token could be extracted and then the vault accessed. It's easier to rotate the token, revoke it and limit access so this could be an acceptable risk to the ISO audit but again it may be considered high risk.
  3. Separately encrypt token before storing in Gitlab CI - As we control the gitlab runners we would then configure these to decrypt this before running. This would mean having some extra scripting and steps in the way which could be a hassle and also another avenue for making a mistake. This seems a bit hacky.
  4. Pass environment credentials on gitlab runner boxes / custom docker images - Again as we control the runners, we could ensure that each of our gitlab runner environments can extract environment variables locally without needing passing from Gitlab. I'm not sure how easy or elegant this would be. As the builds are a mix of shell and docker runners, then it might mean mounting volumes or custom building all containers.
  5. Do no auto-deploy from Gitlab - Have a separate process that is triggered by a webhook from Gitlab on commit and run a completely separate deploy script in full isolation. This is probably the most secure way of doing it in terms of being able to state with certainty. It just means having a separate process which isn't as easy to manage as the Gitlab CI setup.

Obviously there is a decent chance I'm overthinking this but in these early stages of the environment I'm trying to think of all the various weird cases we or others might want to do. We obviously have to trust the Gitlab infrastructure to a degree but it's useful to know if there is a way to not have to rely on that trust while still using the advantages of the easy to use CI system. Something something about having your cake and eating too.