Discussion about supporting pip-tools for dependent projects
As a developer
When I commit changes
I want to know that a known, version-consistent set of python packages will be deployed to production
Problem space
-
Currently in our project, even if we specify one (or more, related) valid requirements.in file for pip-compile, it cannot consider packages that were pre-installed in the docker image, or installed using
pip install packagename
in Dockerfile and the like, and therefore cannot produce requirement.txt file(s) that are guaranteed to install at deploy time without conflicts. -
Pip is now more strict about dependency resolution and while the old way (replace any installed version of a dependency with whatever is needed to install this package right now) can be preserved (--no-deps), this is not solving the problem of broken dependencies, just deferring it until runtime.
-
The version of python could well influence whether a self-consistent requirements file can be created by pip-compile, a great example of this is the pinned requirements/base.txt in the alpine image which dynamically detects the current python version but always installs
google-api-core[grpc] >= 1.14.0, < 2.0.0dev
which worked for a developer once but is not guaranteed to work on every python version or be compatible with a particular project's dependencies. -
Running apt update and pip -U from a stale docker image during deployment/testing adds complication.
Solution space
Ideally, rather than pre-installing pip packages for speed (presumably due to packages that compile), use gitlab's pypi mirror and cache the wheels. This could bring other advantages if done right such as improved network speed, secure package repo or reviewing incoming packages.
Failing that:
-
Make the dockerimage project a python package so that projects can depend on it and therefore pip-compile will fetch it and consider its setup.py file.
-
Only install packages that are absolutely required as part of the devops tools, mandatory logging/monitoring for example. Adding extra packages is not helpful since down the road they or their dependencies may conflict with new project dependencies.
-
Specify your requirements loosely using
install_requires
insetup.py
. -
Avoid depending on packages that have many dependencies or have strict dependency versions.
-
Encourage the use of
pip-sync requirements.txt
to install the requirements instead ofpip install -r requirements.txt
-
Where Alpine images are used, consider slim images instead. Their advantages are no longer assured and their differences can cause issues. One of which is that the alpine pypi mirror is not always aligned or feature-complete with pypi.
References
pip-tools pip is more strict about dependency resolution StackOverflow discussion A dev discusses Alpine