FAQ | This is a LIVE service | Changelog

Pre-commit safety check job fails since safety 3.2.5

👓 What did you see?

Pre-commit jobs failing that are using the safety pre-commit hook and the pre-commit Docker image (for example, through the pre-commit CI template).

Job that failed:

Job that passed:

✅ What did you expect to see?

These jobs not to fail, the same as how these jobs were not failing before the weekend on the exact same code.

💻 Where does this happen?

Jobs that are using the pre-commit CI template, which are using the pre-commit Dockerfile

🔬 How do I recreate this?

% docker run registry.gitlab.developers.cam.ac.uk/uis/devops/infra/dockerimages/pre-commit:3.3 pip install safety==3.2.5
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Collecting safety==3.2.5
  Downloading safety-3.2.5-py3-none-any.whl.metadata (37 kB)
Requirement already satisfied: setuptools>=65.5.1 in /usr/local/lib/python3.11/site-packages (from safety==3.2.5) (65.5.1)
Collecting Click>=8.0.2 (from safety==3.2.5)
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: urllib3>=1.26.5 in /usr/local/lib/python3.11/site-packages (from safety==3.2.5) (2.2.2)
Requirement already satisfied: requests in /usr/local/lib/python3.11/site-packages (from safety==3.2.5) (2.32.3)
Collecting packaging>=21.0 (from safety==3.2.5)
  Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting dparse>=0.6.4b0 (from safety==3.2.5)
  Downloading dparse-0.6.4b0-py3-none-any.whl.metadata (5.5 kB)
Collecting ruamel.yaml>=0.17.21 (from safety==3.2.5)
  Downloading ruamel.yaml-0.18.6-py3-none-any.whl.metadata (23 kB)
Collecting jinja2>=3.1.0 (from safety==3.2.5)
  Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting marshmallow>=3.15.0 (from safety==3.2.5)
  Downloading marshmallow-3.21.3-py3-none-any.whl.metadata (7.1 kB)
Collecting Authlib>=1.2.0 (from safety==3.2.5)
  Downloading Authlib-1.3.1-py2.py3-none-any.whl.metadata (3.8 kB)
Requirement already satisfied: rich in /usr/local/lib/python3.11/site-packages (from safety==3.2.5) (13.7.1)
Collecting typer (from safety==3.2.5)
  Downloading typer-0.12.3-py3-none-any.whl.metadata (15 kB)
Requirement already satisfied: pydantic>=1.10.12 in /usr/local/lib/python3.11/site-packages (from safety==3.2.5) (2.8.2)
Collecting safety-schemas>=0.0.3 (from safety==3.2.5)
  Downloading safety_schemas-0.0.3-py3-none-any.whl.metadata (1.2 kB)
Requirement already satisfied: typing-extensions>=4.7.1 in /usr/local/lib/python3.11/site-packages (from safety==3.2.5) (4.12.2)
Collecting filelock~=3.12.2 (from safety==3.2.5)
  Downloading filelock-3.12.4-py3-none-any.whl.metadata (2.8 kB)
Collecting psutil~=6.0.0 (from safety==3.2.5)
  Downloading psutil-6.0.0.tar.gz (508 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 508.1/508.1 kB 10.4 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting cryptography (from Authlib>=1.2.0->safety==3.2.5)
  Downloading cryptography-43.0.0-cp39-abi3-musllinux_1_2_x86_64.whl.metadata (5.4 kB)
Collecting MarkupSafe>=2.0 (from jinja2>=3.1.0->safety==3.2.5)
  Downloading MarkupSafe-2.1.5-cp311-cp311-musllinux_1_1_x86_64.whl.metadata (3.0 kB)
Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.11/site-packages (from pydantic>=1.10.12->safety==3.2.5) (0.7.0)
Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.11/site-packages (from pydantic>=1.10.12->safety==3.2.5) (2.20.1)
Collecting ruamel.yaml.clib>=0.2.7 (from ruamel.yaml>=0.17.21->safety==3.2.5)
  Downloading ruamel.yaml.clib-0.2.8-cp311-cp311-musllinux_1_1_x86_64.whl.metadata (2.2 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/site-packages (from requests->safety==3.2.5) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/site-packages (from requests->safety==3.2.5) (3.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/site-packages (from requests->safety==3.2.5) (2024.7.4)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.11/site-packages (from rich->safety==3.2.5) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.11/site-packages (from rich->safety==3.2.5) (2.18.0)
Collecting shellingham>=1.3.0 (from typer->safety==3.2.5)
  Downloading shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich->safety==3.2.5) (0.1.2)
Collecting cffi>=1.12 (from cryptography->Authlib>=1.2.0->safety==3.2.5)
  Downloading cffi-1.17.0-cp311-cp311-musllinux_1_1_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.12->cryptography->Authlib>=1.2.0->safety==3.2.5)
  Downloading pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Downloading safety-3.2.5-py3-none-any.whl (171 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 171.9/171.9 kB 18.3 MB/s eta 0:00:00
Downloading Authlib-1.3.1-py2.py3-none-any.whl (223 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 223.8/223.8 kB 20.3 MB/s eta 0:00:00
Downloading click-8.1.7-py3-none-any.whl (97 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 17.0 MB/s eta 0:00:00
Downloading dparse-0.6.4b0-py3-none-any.whl (11 kB)
Downloading filelock-3.12.4-py3-none-any.whl (11 kB)
Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.3/133.3 kB 18.7 MB/s eta 0:00:00
Downloading marshmallow-3.21.3-py3-none-any.whl (49 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.2/49.2 kB 11.2 MB/s eta 0:00:00
Downloading packaging-24.1-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.0/54.0 kB 13.2 MB/s eta 0:00:00
Downloading ruamel.yaml-0.18.6-py3-none-any.whl (117 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.8/117.8 kB 15.6 MB/s eta 0:00:00
Downloading safety_schemas-0.0.3-py3-none-any.whl (29 kB)
Downloading typer-0.12.3-py3-none-any.whl (47 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.2/47.2 kB 9.4 MB/s eta 0:00:00
Downloading MarkupSafe-2.1.5-cp311-cp311-musllinux_1_1_x86_64.whl (33 kB)
Downloading ruamel.yaml.clib-0.2.8-cp311-cp311-musllinux_1_1_x86_64.whl (730 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 730.9/730.9 kB 19.9 MB/s eta 0:00:00
Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Downloading cryptography-43.0.0-cp39-abi3-musllinux_1_2_x86_64.whl (4.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 32.7 MB/s eta 0:00:00
Downloading cffi-1.17.0-cp311-cp311-musllinux_1_1_x86_64.whl (478 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 478.3/478.3 kB 28.5 MB/s eta 0:00:00
Downloading pycparser-2.22-py3-none-any.whl (117 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.6/117.6 kB 17.6 MB/s eta 0:00:00
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml): started
  Building wheel for psutil (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for psutil (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      /tmp/tmpmf6jamsy.c:1:10: fatal error: linux/ethtool.h: No such file or directory
          1 | #include <linux/ethtool.h>
            |          ^~~~~~~~~~~~~~~~~
      compilation terminated.
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_common.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_psaix.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_psbsd.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_compat.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_pslinux.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_pssunos.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_psposix.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_psosx.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_pswindows.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/__init__.py -> build/lib.linux-x86_64-cpython-311/psutil
      creating build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_process_all.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/__main__.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_memleaks.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_process.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_testutils.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_system.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/__init__.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/runner.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      running build_ext
      building 'psutil._psutil_linux' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/psutil
      creating build/temp.linux-x86_64-cpython-311/psutil/arch
      creating build/temp.linux-x86_64-cpython-311/psutil/arch/linux
      gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=600 -DPy_LIMITED_API=0x03060000 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -DPSUTIL_LINUX=1 -I/usr/local/include/python3.11 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-cpython-311/psutil/_psutil_common.o
      gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=600 -DPy_LIMITED_API=0x03060000 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -DPSUTIL_LINUX=1 -I/usr/local/include/python3.11 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-cpython-311/psutil/_psutil_linux.o
      psutil/_psutil_linux.c:13:10: fatal error: linux/ethtool.h: No such file or directory
         13 | #include <linux/ethtool.h>  // DUPLEX_*
            |          ^~~~~~~~~~~~~~~~~
      compilation terminated.
      psutil could not be installed from sources. Perhaps Python header files are not installed. Try running:
        sudo apk add gcc python3-dev musl-dev linux-headers
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for psutil
Failed to build psutil
ERROR: Could not build wheels for psutil, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip

📚 Any additional information?

See comment https://gitlab.developers.cam.ac.uk/uis/devops/gsuite/preferences-webapp/-/issues/55#note_673883.

I think I narrowed it down to this commit in the safety library:

https://github.com/pyupio/safety/commit/83a07ec7360efdcd39962a816b9affc8ab313dde

This was released on the 9th of August in their 3.2.5 version release.

The fix is by making sure linux-headers is installed as a package in the pre-commit container.