This makes sense to me, and I definitely like the fact that we have a means of emulating the async tasks for local development. My only thought was that we'd potentially want to use Cloud Tasks over PubSub, given that Tasks is more designed for what we need here (https://cloud.google.com/pubsub/docs/choosing-pubsub-or-cloud-tasks).

Also, there's probably a separate MR for this - but the CELERYBEAT_SCHEDULE tasks would need to move over to being endpoints triggered by Cloud Scheduler.

I tend to agree with Monty on going with Cloud Tasks given the following:

1. Cloud Tasks supports explicit invocation which is one of the features we'd like to use in the webapp async tasks. For instance, once the webapp validates a request for IFS storage it should immediately start provisioning the space. PubSub in this scenario will leave the invocation "decision" to the subscription service (implicit invocation) which gives little control to our process.
2. PubSub has a configuration called Acknowledgement deadline which has a maximum of 10 minutes. This is potentially not enough for particular NetApp tasks which could take more than that (e.g., data resync between two storage VMs)

I have opened two MR to evaluate both options.

Yes, Monty, Cloud Scheduler tasks setup are in a different MR.

That sounds sensible to me. Cloud Tasks supports HTTP endpoints which means we can use substantially the same approach. (I don't think we want to try and make SSGW work in AppEngine(!)) In particular Cloud Tasks supports the same authentication scheme as described above.

There is no Google-blessed Cloud Tasks emulator for local development but there is a third party one which might meet our needs.