Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[universal] Conda: patch Python due to CVE-2023-32681 and GHSA-5cpq-8wj7-hf2v #627

Merged
merged 26 commits into from
Aug 2, 2023
Merged

[universal] Conda: patch Python due to CVE-2023-32681 and GHSA-5cpq-8wj7-hf2v #627

merged 26 commits into from
Aug 2, 2023

Conversation

alexander-smolyakov
Copy link
Contributor

@alexander-smolyakov alexander-smolyakov commented Jun 15, 2023

Dev container name:

  • universal

Issue description:

This PR aims to address CVE-2023-32681 and GHSA-5cpq-8wj7-hf2v security vulnerabilities. These vulnerabilities come from Python distribution which ships with Conda.

Fix description:

In the universal devcontainer, we have three instances of Python distribution:

flowchart TD
    CF[conda feature] --> CFP[Python 3.10.4]
    PF[python feature] --> PFP1[Python 3.10.8]
    PF --> PFP2[Python 3.9.16]
    CFP --"/opt/conda/bin/python3"--> U[universal devconatiner]
    PFP1 --"/usr/local/python/3.10.8/bin/python (set to current)"--> U[universal devconatiner]
    PFP2 --"/usr/local/python/3.9.16/bin/python"--> U[universal devconatiner]
Loading

Originally fixes were applied during the setup-user feature installation and the feature installation queue looks the following:

flowchart TD
   D[Dockerfile] --"..."--> PF[python feature]
   PF --> PPF[patch-python feature] 
   PPF --> MLF[machine-learning-packages feature]
   MLF --"..." --> CF[conda feature]
   CF --"..." --> SU[setup-user feature]
Loading

This approach resulted in the Anaconda Python distribution containing vulnerable packages in site-packages folders, and this triggered our security checks. Additionally, when running the conda list command, it's listed vulnerable packages.

The pip doesn't provide the ability to remove packages from the specific site-packages folder. When we try to remove any packages for related Python distribution during setup-user installation, we face the following situation:

codespace ➜ / $ /opt/conda/bin/python3 -m pip uninstall requests
Found existing installation: requests 2.31.0
Uninstalling requests-2.31.0:
  Would remove:
    /home/codespace/.local/lib/python3.10/site-packages/requests-2.31.0.dist-info/*
    /home/codespace/.local/lib/python3.10/site-packages/requests/*
Proceed (Y/n)?

The pip trying to remove the package from the /home/codespace/.local/lib/python3.10/site-packages/ folder when we need to remove the package from opt/conda/lib/python3.10/site-packages. In order to remove package opt/conda/lib/python3.10/site-packages, we should run the /opt/conda/bin/python3 -m pip uninstall requests command twice because we installed additional Python distribution via the python feature:

codespace ➜ / $ /opt/conda/bin/python3 -m pip uninstall requests
Found existing installation: requests 2.31.0
Uninstalling requests-2.31.0:
  Would remove:
    /home/codespace/.local/lib/python3.10/site-packages/requests-2.31.0.dist-info/*
    /home/codespace/.local/lib/python3.10/site-packages/requests/*
Proceed (Y/n)? y
  Successfully uninstalled requests-2.31.0
codespace ➜ / $ /opt/conda/bin/python3 -m pip uninstall requests
Found existing installation: requests 2.28.1
Uninstalling requests-2.28.1:
  Would remove:
    /opt/conda/lib/python3.10/site-packages/requests-2.28.1.dist-info/*
    /opt/conda/lib/python3.10/site-packages/requests/*
Proceed (Y/n)? y
  Successfully uninstalled requests-2.28.1
codespace ➜ / $ /opt/conda/bin/python3 -m pip uninstall requests
WARNING: Skipping requests as it is not installed.

The other side effect of this approach is that it affects Python distribution coming from the python feature:

codespace ➜ / $ which python
/home/codespace/.python/current/bin/python
codespace ➜ / $ python --version
Python 3.10.8
codespace ➜ / $ python -c "import requests; print(requests.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'

To avoid such a situation, we have to reorganize the features installation queue to install the conda feature and apply all patches to it before the python feature installation.

Fix results:

Before fix:

opt/conda/lib/python3.10/site-packages
root ➜ / $ ls -la opt/conda/lib/python3.10/site-packages
total 3040
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 .
drwxrwsrwx 1 codespace oryx   12288 Jun 13 00:09 ..
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 OpenSSL
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 PySocks-1.7.1.dist-info
-rw-rw-r-- 1 codespace oryx     119 Mar 31  2022 README.txt
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 __pycache__
-rwxrwxr-x 1 codespace oryx  663864 Jan 21  2022 _cffi_backend.cpython-310-x86_64-linux-gnu.so
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 _distutils_hack
-rwxrwxr-x 1 codespace oryx 1765544 Jan  7  2022 _ruamel_yaml.cpython-310-x86_64-linux-gnu.so
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 brotli
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 brotlipy-0.7.0-py3.10.egg-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 certifi
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 certifi-2022.12.7.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 cffi
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 cffi-1.15.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 charset_normalizer
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 charset_normalizer-2.0.4.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 conda
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 conda-23.1.0-py3.10.egg-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 conda_env
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 conda_package_handling
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 conda_package_handling-2.0.2.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 conda_package_streaming
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 conda_package_streaming-0.7.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 cryptography
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 cryptography-39.0.1.dist-info
-rw-rw-r-- 1 codespace oryx     151 Jan  5 17:37 distutils-precedence.pth
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 idna
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 idna-3.4.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 pip
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 pip-23.0.1-py3.10.egg-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 pkg_resources
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 pluggy
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 pluggy-1.0.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 pyOpenSSL-23.0.0.dist-info
-rw-rw-r-- 1 codespace oryx    6371 Dec 29  2021 pycosat-0.6.3-py3.10.egg-info
-rwxrwxr-x 1 codespace oryx  384544 Dec 29  2021 pycosat.cpython-310-x86_64-linux-gnu.so
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 pycparser
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 pycparser-2.21.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 requests
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 requests-2.28.1.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 ruamel
-rw-rw-r-- 1 codespace oryx     539 Jan  7  2022 ruamel.yaml-0.16.10-py3.10-nspkg.pth
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 ruamel.yaml-0.16.10.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 ruamel.yaml.clib-0.2.6.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 setuptools
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 setuptools-65.6.3-py3.10.egg-info
-rw-rw-r-- 1 codespace oryx   31086 Dec 29  2021 socks.py
-rw-rw-r-- 1 codespace oryx    3966 Dec 29  2021 sockshandler.py
-rw-rw-r-- 1 codespace oryx    8806 Dec 29  2021 test_pycosat.py
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 tlz
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 toolz
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 toolz-0.12.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 tqdm
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 tqdm-4.65.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 urllib3
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 urllib3-1.26.15.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 wheel
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 wheel-0.38.4-py3.10.egg-info
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 xontrib
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 zstandard
drwxrwsrwx 1 codespace oryx    4096 Jun 13 00:09 zstandard-0.15.2.dist-info
conda list
root ➜ / $ conda list
# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
brotlipy                  0.7.0           py310h7f8727e_1002
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2023.01.10           h06a4308_0
certifi                   2022.12.7       py310h06a4308_0
cffi                      1.15.0          py310hd667e15_1
charset-normalizer        2.0.4                    pypi_0    pypi
conda                     23.1.0          py310h06a4308_0
conda-package-handling    2.0.2           py310h06a4308_0
conda-package-streaming   0.7.0           py310h06a4308_0
cryptography              39.0.1          py310h9ce1e76_0
idna                      3.4             py310h06a4308_0
ld_impl_linux-64          2.38                 h1181459_1
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.1.0                hdf63c60_0
libstdcxx-ng              9.1.0                hdf63c60_0
libuuid                   1.0.3                h7f8727e_2
ncurses                   6.3                  h7f8727e_2
openssl                   1.1.1t               h7f8727e_0
pip                       23.0.1          py310h06a4308_0
pluggy                    1.0.0           py310h06a4308_1
pycosat                   0.6.3           py310h7f8727e_0
pycparser                 2.21                     pypi_0    pypi
pyopenssl                 23.0.0          py310h06a4308_0
pysocks                   1.7.1           py310h06a4308_0
python                    3.10.4               h12debd9_0
readline                  8.1.2                h7f8727e_1
requests                  2.28.1          py310h06a4308_1
ruamel.yaml               0.16.10         py310h7f8727e_1
ruamel.yaml.clib          0.2.6           py310h7f8727e_0
setuptools                65.6.3          py310h06a4308_0
sqlite                    3.38.5               hc218d9a_0
tk                        8.6.12               h1ccaba5_0
toolz                     0.12.0          py310h06a4308_0
tqdm                      4.65.0          py310h2f386ee_0
tzdata                    2022g                h04d1e81_0
urllib3                   1.26.15         py310h06a4308_0
wheel                     0.38.4          py310h06a4308_0
xz                        5.2.5                h7f8727e_1
zlib                      1.2.12               h7f8727e_2
zstandard                 0.15.2          py310h7f8727e_0

With fix:

opt/conda/lib/python3.10/site-packages
codespace ➜ / $ ls -la opt/conda/lib/python3.10/site-packages
total 3248
drwxrwsrwx 1 codespace oryx    4096 Jun 15 09:11 .
drwxrwsrwx 1 codespace oryx   12288 Jun 14 15:17 ..
drwxrwsr-x 1 codespace oryx    4096 Jun 15 09:11 OpenSSL
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 PySocks-1.7.1.dist-info
-rw-rw-r-- 1 codespace oryx     119 Mar 31  2022 README.txt
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 __pycache__
-rwxrwxr-x 1 codespace oryx  663864 Jan 21  2022 _cffi_backend.cpython-310-x86_64-linux-gnu.so
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 _distutils_hack
-rwxrwxr-x 1 codespace oryx 1765544 Jan  7  2022 _ruamel_yaml.cpython-310-x86_64-linux-gnu.so
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 brotli
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 brotlipy-0.7.0-py3.10.egg-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 certifi
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 certifi-2022.12.7.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 cffi
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 cffi-1.15.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 charset_normalizer
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 charset_normalizer-2.0.4.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 conda
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 conda-23.1.0-py3.10.egg-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 conda_env
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 conda_package_handling
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 conda_package_handling-2.0.2.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 conda_package_streaming
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 conda_package_streaming-0.7.0.dist-info
drwxrwsr-x 1 codespace oryx    4096 Jun 15 09:11 cryptography
drwxrwsr-x 1 codespace oryx    4096 Jun 15 09:11 cryptography-41.0.1.dist-info
-rw-rw-r-- 1 codespace oryx     151 Jan  5 17:37 distutils-precedence.pth
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 idna
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 idna-3.4.dist-info
drwxrwsr-x 1 codespace oryx    4096 Jun 15 09:11 pip
drwxrwsr-x 1 codespace oryx    4096 Jun 15 09:11 pip-23.1.2.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 pkg_resources
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 pluggy
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 pluggy-1.0.0.dist-info
drwxrwsr-x 1 codespace oryx    4096 Jun 15 09:11 pyOpenSSL-23.2.0.dist-info
-rw-rw-r-- 1 codespace oryx    6371 Dec 29  2021 pycosat-0.6.3-py3.10.egg-info
-rwxrwxr-x 1 codespace oryx  384544 Dec 29  2021 pycosat.cpython-310-x86_64-linux-gnu.so
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 pycparser
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 pycparser-2.21.dist-info
drwxrwsr-x 1 codespace oryx    4096 Jun 15 09:11 requests
drwxrwsr-x 1 codespace oryx    4096 Jun 15 09:11 requests-2.31.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 ruamel
-rw-rw-r-- 1 codespace oryx     539 Jan  7  2022 ruamel.yaml-0.16.10-py3.10-nspkg.pth
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 ruamel.yaml-0.16.10.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 ruamel.yaml.clib-0.2.6.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 setuptools
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 setuptools-65.6.3-py3.10.egg-info
-rw-rw-r-- 1 codespace oryx   31086 Dec 29  2021 socks.py
-rw-rw-r-- 1 codespace oryx    3966 Dec 29  2021 sockshandler.py
-rw-rw-r-- 1 codespace oryx    8806 Dec 29  2021 test_pycosat.py
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 tlz
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 toolz
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 toolz-0.12.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 tqdm
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 tqdm-4.65.0.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 urllib3
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 urllib3-1.26.15.dist-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 wheel
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 wheel-0.38.4-py3.10.egg-info
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 xontrib
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 zstandard
drwxrwsrwx 1 codespace oryx    4096 Jun 14 15:17 zstandard-0.15.2.dist-info
conda list
codespace ➜ / $ conda list
# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
brotlipy                  0.7.0           py310h7f8727e_1002  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2023.01.10           h06a4308_0  
certifi                   2022.12.7       py310h06a4308_0  
cffi                      1.15.0          py310hd667e15_1  
charset-normalizer        2.0.4                    pypi_0    pypi
conda                     23.1.0          py310h06a4308_0  
conda-package-handling    2.0.2           py310h06a4308_0  
conda-package-streaming   0.7.0           py310h06a4308_0  
cryptography              41.0.1                   pypi_0    pypi
idna                      3.4             py310h06a4308_0  
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libuuid                   1.0.3                h7f8727e_2  
ncurses                   6.3                  h7f8727e_2  
openssl                   1.1.1t               h7f8727e_0  
pip                       23.1.2                   pypi_0    pypi
pluggy                    1.0.0           py310h06a4308_1  
pycosat                   0.6.3           py310h7f8727e_0  
pycparser                 2.21                     pypi_0    pypi
pyopenssl                 23.2.0                   pypi_0    pypi
pysocks                   1.7.1           py310h06a4308_0  
python                    3.10.4               h12debd9_0  
readline                  8.1.2                h7f8727e_1  
requests                  2.31.0                   pypi_0    pypi
ruamel.yaml               0.16.10         py310h7f8727e_1  
ruamel.yaml.clib          0.2.6           py310h7f8727e_0  
setuptools                65.6.3          py310h06a4308_0  
sqlite                    3.38.5               hc218d9a_0  
tk                        8.6.12               h1ccaba5_0  
toolz                     0.12.0          py310h06a4308_0  
tqdm                      4.65.0          py310h2f386ee_0  
tzdata                    2022g                h04d1e81_0  
urllib3                   1.26.15         py310h06a4308_0  
wheel                     0.38.4          py310h06a4308_0  
xz                        5.2.5                h7f8727e_1  
zlib                      1.2.12               h7f8727e_2  
zstandard                 0.15.2          py310h7f8727e_0  

Changelog:

  • Reorganized features installation queue;
  • Implemented patches to bump up versions for the following packages:
  • Introduced patch-conda feature.

Checklist:

  • Checked that applied changes work as expected

…wj7-hf2v

- Reorganize features installation queue;
- Introduce patch-conda feature;
@alexander-smolyakov alexander-smolyakov requested a review from a team as a code owner June 15, 2023 13:39
Copy link
Member

@samruddhikhandale samruddhikhandale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one comment, can you also help resolve the merge conflicts? thanks!

@azure-pipelines
Copy link

There was an error handling pipeline event c9650c9f-36bc-49b6-9c6f-b01bfaa947a7.

@samruddhikhandale samruddhikhandale merged commit ec6f6a9 into devcontainers:main Aug 2, 2023
3 checks passed
@alexander-smolyakov alexander-smolyakov deleted the users/alexander-smolyakov/universal_GHSA-5cpq-8wj7-hf2v_bump-cryptography-version branch August 2, 2023 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants