Intermittent unavailability while using gitlab as Terraform state file store

Hi,

I'm using Gitlab to manage Terraform state. I've been facing an intermittent problem while using that.

After successfully storing the state file using the Gitlab backend, state file will be retrieved empty a couple of times before returning with the right content.

I'm able to reproduce that with the following.

# Defining gitlab target project and setting up credentials
export GITLAB_PROJECT_ID=2832
export
TF_ADDRESS=https://gitlab.developers.cam.ac.uk/api/v4/projects/${GITLAB_PROJECT_ID}/terraform/state/mydebug
export TF_USERNAME=gb637
export TF_VAR_GITLAB_TOKEN=<PERSONAL_TOKEN>

# Terraform setup. Using http backend and a dummy resource
cat <<EOF > main.tf
terraform {
  backend "http" {
  }
}

resource "null_resource" "sleep" {
  provisioner "local-exec" {
    command = "sleep 1"
  }
}
EOF

# Initiating terraform 
terraform init \
  -upgrade=true \
  -backend-config=address=${TF_ADDRESS} \
  -backend-config=lock_address=${TF_ADDRESS}/lock \
  -backend-config=unlock_address=${TF_ADDRESS}/lock \
  -backend-config=username=${TF_USERNAME} \
  -backend-config=password=${TF_VAR_GITLAB_TOKEN} \
  -backend-config=lock_method=POST \
  -backend-config=unlock_method=DELETE \
  -backend-config=retry_wait_min=5

# Creating, appllying and submiting terraform state
terraform apply -auto-approve

After doing the above I expect to always get the same state file from the API, but that isn't happening.

alias tfm=terraform

10:43 ➜  tfm-debug  tfm state pull
{
  "version": 4,
  "terraform_version": "0.13.4",
  "serial": 0,
  "lineage": "",
  "outputs": {},
  "resources": []
}

10:43 ➜  tfm-debug  tfm state pull
{
  "version": 4,
  "terraform_version": "0.13.4",
  "serial": 0,
  "lineage": "",
  "outputs": {},
  "resources": []
}

10:44 ➜  tfm-debug  tfm state pull
{
  "version": 4,
  "terraform_version": "0.13.4",
  "serial": 0,
  "lineage": "",
  "outputs": {},
  "resources": []
}

10:44 ➜  tfm-debug  tfm state pull
{
  "version": 4,
  "terraform_version": "0.13.4",
  "serial": 1,
  "lineage": "50782fb4-b561-f937-84c6-fa3ae79781e2",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "null_resource",
      "name": "sleep",
      "provider": "provider[\"registry.terraform.io/hashicorp/null\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "9038718454041431944",
            "triggers": null
          },
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

Notice that the state file came in the 4th execution. I followed the same steps using gitlab.com and everything is working as expected.

Seems to me that terraform state store has been configured to point to local gitlab web node directory instead of a shared folder, and I'm hitting distinct web node serving its own local directory.

I hope everything reported here has been helpful to detect and correct this problem. I'm fancy to provide more details or perform more test if you need.

Kind regards