Fix dependency scanning

We are missing dependency scanning checks of python packages from AutoDevOps on all Python apps.

The Dependency Scanning of AutoDevOps requires PIP_REQUIREMENTS_FILE to be set if your are not using a requirements.txt file. See https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html#configuring-specific-analyzers-used-by-dependency-scanning

Most of our repos use requirements/XXX.txt files and a requirements.txt in the root that imports those via -r. -r is not supported by gemnasium, see http://docs.gemnasium.net/en/stable/languages/python.html

changed milestone to %DevOps Sprint 130

added issuetypeBug priority1 High + 1 deleted label

Sizing as 2d spike. If it's not fixed by then, re-group and discuss.

changed time estimate to 2d

added spike workflowSprint Ready labels and removed 1 deleted label

set weight to 2

added teamCloud label

changed milestone to %DevOps Sprint 129

changed milestone to %DevOps Sprint 130

changed milestone to %DevOps Sprint 131

assigned to @rjg21

added workflowIn Progress label and removed workflowSprint Ready label

I believe I may have just spend some time trying to fix something that is no longer a problem.

We are setting DS_ANALYZER_IMAGE in auto-devops/custom-gemnasium.yml (which gets included in the auto-devops/tox-tests.yml) to use our custom gemnasium image in https://gitlab.developers.cam.ac.uk/uis/devops/infra/dockerimages . This hasn't been build it ages as they swapped to slim and broken our customisation (that I don't believe we need anymore anyway).

However, looking recent pipeline jobs this var is being ignored and the gemnasium from gitlab being used. Additionally, it appears that it is finding the requirements in all the -r includes on the dependency report.

So, putting this up for review, for another set of eye to confirm that I've not gone mad. If this is no longer an issue then clear up of this and dockerimages repo can be done.

added workflowReview Required label and removed workflowIn Progress label

Another example of a django app that is being updated at the moment:

Pipeline - https://gitlab.developers.cam.ac.uk/uis/devops/research-dashboard/webapp/-/jobs/384198 (though oddly a gemnasium-dependency_scanning job not gemnasium-python)

Dependency report - https://gitlab.developers.cam.ac.uk/uis/devops/research-dashboard/webapp/-/dependencies (requirements.txt discoveries at end and 2 days ago)

So auto-devops/custom-gemnasium.yml sets DS_ANALYZER_IMAGES not DS_ANALYZER_IMAGE. The former isn't used by Gitlabs AutoDevOps the latter would break the other (non-python) analysers in Auto-Devops Dependency CI.

The AutoDevOps gemnasium-python job has a rule the makes it run if it finds requirements.txt files even in subfolders. However, it also checks $CI_COMMIT_BRANCH which means it only runs on branch commits not MRs. Hence it not running in the Research Dashboard pipeline linked above but does in a merge to master pipeline.

I believe (not a go programmer) that the dependency list builder now uses pip to get the dependencies so -r will be included.

If you're happy that this is not longer an issue, I'll create issues to clean up:

• dockerimages custom gemnastium image
• ci-templates use of custom-gemnasium.yml

Seems sensible to me. And indeed the dependency list builder does appear to use pip so happy for this to be closed if the issues above are opened.

Moving to ~"workflow::rework" to create issues.

Both merge requests opened. Moving back to ~"workflow::review required"

MR to CI template needs version stamping and changelog. Dockerimages removal merged.

done

will tag once merged

changed milestone to %DevOps Sprint 132

assigned to @amc203

added workflowRework label and removed workflowReview Required label

mentioned in merge request uis/devops/infra/dockerimages!66 (merged)