Load scope definitions dynamically

Related to https://gitlab.developers.cam.ac.uk/uis/devops/api/gateway-ops/-/issues/75 on the Gateway, we would like a more automated way to describe scopes. As part of https://gitlab.developers.cam.ac.uk/uis/devops/api/gateway-ops/-/issues/75 we will probably end up with a mechanism to specify supported scopes alongside a product which we should design in a way which allows re-use by this application.

changed milestone to %Backlog

added workflowNeeds Refinement label

mentioned in commit d4bdf5d1

mentioned in merge request !1 (merged)

mentioned in commit e02711e0

mentioned in commit c8504fb5

mentioned in commit fe161779

mentioned in commit 4c935b65

mentioned in commit 60158aa2

mentioned in commit 8b481171

changed milestone to %DevOps Sprint 131

This is kinda on the critical path for the Card Frontend - as we need to give users the ability to consent to scopes defined in the Card API (as implemented under https://gitlab.developers.cam.ac.uk/uis/devops/iam/card-database/card-api/-/issues/116).

We could hard-code the scopes in this app rather than auto-discover them. It's not nice but it'd free up the Card API.

Good point - I've opened #7 (closed) to track this.

added priority2 Medium label

mentioned in issue #7 (closed)

Discussed in planning:

• The best means of doing this would be to fetch the scopes from the OpenAPI specification - which would require that the scopes are well documented in the OpenAPI spec, and also moves us towards update and publish the OpenAPI specifications automatically in the developer portal,
• As a pre-requisite we should ask our Apigee contact about the best approach for this, along with the best way to manage OpenAPI specifications,

We need to design a good approach to both managing scopes and openapi specs, so sizing this as a spike.

Dropping https://pypi.org/project/openapi3/ and https://pypi.org/project/prance/ as good libraries to use in any automation.

I've had a poke in https://apidocs.apigee.com/apis and I can't see any obvious APIs for managing specs in the portal. I'll poke the Apigee contact.

Mail sent:

Hi all,

We're productising our Apigee instance over at the University of Cambridge and are looking to firm up some of our Apigee-side checking. In particular we want to make sure we enforce the correct OAuth2 scopes for each path/method and would like to make use of the OASValidation policy.

Both require we have an OpenAPI spec for each API which isn't an issue but we're running into a problem where the specs in the portal associated with each product in the catalogue can drift out of sync with the most recent specs.

So a couple of questions:

a) Can we automate updating specs in the API catalogues? I had a poke in https://apidocs.apigee.com/apis and there doesn't appear to be much related to that area.

b) Can we fetch an OpenAPI spec dynamically (with caching) for OASValidation. We'd rather that people publishing APIs through our instance not have to request we deploy a new instance of the proxy whenever they update their backend.

Thanks in advance,

Reply:

Hi Rich,

For a) We are working on the 'spec API' for automated publishing of the API Specs to the Integrated Developer portal - we are expecting that to drop in late Q4, but holiday moratoriums may push the release out until early January.

There is a private preview for an Alpha version of the spec API, but it will change, including breaking changes as it moves forward to release. I can request for you to be added to the private preview if you want, or alternatively if you can live with the late Q4 / early Jan timeline then I would suggest waiting for the proper API for to drop.

For b) then unfortunately not, the OAS spec needs to be added as a proxy resource and that will require a redeployment. I'll ping our PM team and see if there are any plans to make this more flexible.

Best Regards,

Nigel

It'd be nice to get a bit more detail about what the spec API looks like and what functionality it enables. Can we be added to the alpha and try it out on our test Apigee instance? We'll likely still need to build on top of it in order to be able to publish scopes into the login and consent app, but finding out more about the functionality the API gives us would help us plan any custom developer work.

The "late Q4 / early Jan timeline" coming to an end, and no sign of a spec API in their release notes. Guess this issue waits a bit longer...

I think we discussed at planning that what we'd do for the time being is commit the openapi specs which are published behind the portal within the gateway-ops repo, and then traverse the scope definitions within those specs at apply-time in order to create the map of scopes which are used by the login and consent app.

This is definitely a workaround, but I don't think we can do anything better until Apigee come up with a solution.

Having the OpenAPI specs in source control is also useful if we go down the OPA route (see comments on https://gitlab.developers.cam.ac.uk/uis/devops/iam/identity-business-analysis/-/issues/13) since we can write policies which reference information from the OpenAPI specs directly.

Wondering if this issue needs to go back to ~"workflow::needs refinement" for next sprint as I'm not sure it is clear what piece of work this issue covers from the discussion above, gateway-ops#75 and uis/devops/iam/identity-business-analysis#13.

I believe that the aim of this story is not to worry about enforcing scopes, or making use of them within any authentication layer at the API Gateway level, but instead to populate the scopes which are currently hardcoded within the login and consent app (https://gitlab.developers.cam.ac.uk/uis/devops/api/login-and-consent-webapp/-/blob/main/login/scopes.py) from the OpenAPI specs which are currently within the API Gateway's portal.

I believe the work would consist of:

• Update the login and consent app to load the scopes from external settings rather than the hard-coded set defined in https://gitlab.developers.cam.ac.uk/uis/devops/api/login-and-consent-webapp/-/blob/main/login/scopes.py,
• Add all OpenAPI specs which are currently published within the API Gateway's developer portal (https://apigee.google.com/organizations/api-prod-a3dc87f7/portals/api-prod-a3dc87f7-prod/api-catalog/apis) to the Gateway Ops repo,
• Update the terraform within Gateway ops to iterate through each OpenAPI spec and populate the login and consent app's external settings with the scope definitions, formatted as required (this might take some tweaking of the existing OpenAPI specs so that the scope descriptions make sense). I'd probably use an external data source to implement this so I could do it in Python, but that's because HCL scares me,

Does that make sense? Apologies the issue description wasn't updated after we discussed the issue in sprint planning. Happy to chat it through.

Excellent summary.

I've picked this up to have a first pass proposal for something concrete. For the moment I'll just concentrate on the Lookup and Card APIs as examples.

changed time estimate to 2d

added spike workflowSprint Ready labels and removed workflowNeeds Refinement label

set weight to 2

added teamIdentity label

changed milestone to %DevOps Sprint 132