Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Python 3.12.x (#5736) #5951

Open
wants to merge 15 commits into
base: develop
Choose a base branch
from

Conversation

dsotirho-ucsc
Copy link
Contributor

@dsotirho-ucsc dsotirho-ucsc commented Feb 10, 2024

Connected issues: #5736

Checklist

Author

  • PR is a draft
  • Target branch is develop
  • Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
  • On ZenHub, PR is connected to all issues it (partially) resolves
  • PR description links to connected issues
  • PR title matches1 that of a connected issue or comment in PR explains why they're different
  • PR title references all connected issues
  • For each connected issue, there is at least one commit whose title references that issue

1 when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

  • Added p tag to titles of partial commits
  • This PR is labeled partial or completely resolves all connected issues
  • This PR partially resolves each of the connected issues or does not have the partial label

Author (chains)

  • This PR is blocked by previous PR in the chain or is not chained to another PR
  • The blocking PR is labeled base or this PR is not chained to another PR
  • This PR is labeled chained or is not chained to another PR

Author (reindex, API changes)

  • Added r tag to commit title or the changes introduced by this PR will not require reindexing of any deployment
  • This PR is labeled reindex:dev or the changes introduced by it will not require reindexing of dev
  • This PR is labeled reindex:anvildev or the changes introduced by it will not require reindexing of anvildev
  • This PR is labeled reindex:anvilprod or the changes introduced by it will not require reindexing of anvilprod
  • This PR is labeled reindex:prod or the changes introduced by it will not require reindexing of prod
  • This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod
  • This PR and its connected issues are labeled API or this PR does not modify a REST API
  • Added a (A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST API
  • Updated REST API version number in app.py or this PR does not modify a REST API

Author (upgrading deployments)

  • Ran make docker_images.json and committed the resulting changes or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable
  • Documented upgrading of deployments in UPGRADING.rst or this PR does not require upgrading deployments
  • Added u tag to commit title or this PR does not require upgrading deployments
  • This PR is labeled upgrade or does not require upgrading deployments
  • This PR is labeled deploy:shared or does not modify docker_images.json, and does not require deploying the shared component for any other reason
  • This PR is labeled deploy:gitlab or does not require deploying the gitlab component
  • This PR is labeled deploy:runner or does not require deploying the runner image

Author (hotfixes)

  • Added F tag to main commit title or this PR does not include permanent fix for a temporary hotfix
  • Reverted the temporary hotfixes for any connected issues or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues connected to this PR

Author (before every review)

  • Rebased PR branch on develop, squashed old fixups
  • Ran make requirements_update or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile
  • Added R tag to commit title or this PR does not modify requirements*.txt
  • This PR is labeled reqs or does not modify requirements*.txt
  • make integration_test passes in personal deployment or this PR does not modify functionality that could affect the IT outcome

Peer reviewer (after approval)

  • PR is not a draft
  • Ticket is in Review requested column
  • PR is awaiting requested review from system administrator
  • PR is assigned to only the system administrator

System administrator (after approval)

  • Actually approved the PR
  • Labeled connected issues as demo or no demo
  • Commented on connected issues about demo expectations or all connected issues are labeled no demo
  • Decided if PR can be labeled no sandbox
  • A comment to this PR details the completed security design review
  • PR title is appropriate as title of merge commit
  • N reviews label is accurate
  • Moved connected issues to Approved column
  • PR is assigned to only the operator

Operator (before pushing merge the commit)

  • Checked reindex:… labels and r commit title tag
  • Checked that demo expectations are clear or all connected issues are labeled no demo
  • Squashed PR branch and rebased onto develop
  • Sanity-checked history
  • Pushed PR branch to GitHub
  • Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused or this PR is not labeled deploy:shared
  • Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply or this PR is not labeled deploy:gitlab
  • Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused or this PR is not labeled deploy:shared
  • Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply or this PR is not labeled deploy:gitlab
  • Checked the items in the next section or this PR is labeled deploy:gitlab
  • PR is assigned to only the system administrator or this PR is not labeled deploy:gitlab

System administrator

  • Background migrations for dev.gitlab are complete or this PR is not labeled deploy:gitlab
  • Background migrations for anvildev.gitlab are complete or this PR is not labeled deploy:gitlab
  • PR is assigned to only the operator

Operator (before pushing merge the commit)

  • Ran _select dev.gitlab && make -C terraform/gitlab/runner or this PR is not labeled deploy:runner
  • Ran _select anvildev.gitlab && make -C terraform/gitlab/runner or this PR is not labeled deploy:runner
  • Added sandbox label or PR is labeled no sandbox
  • Pushed PR branch to GitLab dev or PR is labeled no sandbox
  • Pushed PR branch to GitLab anvildev or PR is labeled no sandbox
  • Build passes in sandbox deployment or PR is labeled no sandbox
  • Build passes in anvilbox deployment or PR is labeled no sandbox
  • Reviewed build logs for anomalies in sandbox deployment or PR is labeled no sandbox
  • Reviewed build logs for anomalies in anvilbox deployment or PR is labeled no sandbox
  • Deleted unreferenced indices in sandbox or this PR does not remove catalogs or otherwise causes unreferenced indices in dev
  • Deleted unreferenced indices in anvilbox or this PR does not remove catalogs or otherwise causes unreferenced indices in anvildev
  • Started reindex in sandbox or this PR is not labeled reindex:dev
  • Started reindex in anvilbox or this PR is not labeled reindex:anvildev
  • Checked for failures in sandbox or this PR is not labeled reindex:dev
  • Checked for failures in anvilbox or this PR is not labeled reindex:anvildev
  • The title of the merge commit starts with the title of this PR
  • Added PR # reference to merge commit title
  • Collected commit title tags in merge commit title but only included p if the PR is also labeled partial
  • Moved connected issues to Merged lower column in ZenHub
  • Moved blocked issues to Triage or no issues are blocked on the connected issues
  • Pushed merge commit to GitHub

Operator (chain shortening)

  • Changed the target branch of the blocked PR to develop or this PR is not labeled base
  • Removed the chained label from the blocked PR or this PR is not labeled base
  • Removed the blocking relationship from the blocked PR or this PR is not labeled base
  • Removed the base label from this PR or this PR is not labeled base

Operator (after pushing the merge commit)

  • Pushed merge commit to GitLab dev
  • Pushed merge commit to GitLab anvildev
  • Build passes on GitLab dev
  • Reviewed build logs for anomalies on GitLab dev
  • Build passes on GitLab anvildev
  • Reviewed build logs for anomalies on GitLab anvildev
  • Ran _select dev.shared && make -C terraform/shared apply or this PR is not labeled deploy:shared
  • Ran _select anvildev.shared && make -C terraform/shared apply or this PR is not labeled deploy:shared
  • Deleted PR branch from GitHub
  • Deleted PR branch from GitLab dev
  • Deleted PR branch from GitLab anvildev

Operator (reindex)

  • Deindexed all unreferenced catalogs in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Deindexed all unreferenced catalogs in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Deindexed specific sources in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Deindexed specific sources in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Indexed specific sources in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Indexed specific sources in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Started reindex in dev or this PR does not require reindexing dev
  • Started reindex in anvildev or this PR does not require reindexing anvildev
  • Checked for, triaged and possibly requeued messages in both fail queues in dev or this PR does not require reindexing dev
  • Checked for, triaged and possibly requeued messages in both fail queues in anvildev or this PR does not require reindexing anvildev
  • Emptied fail queues in dev or this PR does not require reindexing dev
  • Emptied fail queues in anvildev or this PR does not require reindexing anvildev

Operator

  • Propagated the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels to the next promotion PRs or this PR carries none of these labels
  • Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labels
  • PR is assigned to no one

Shorthand for review comments

  • L line is too long
  • W line wrapping is wrong
  • Q bad quotes
  • F other formatting problem

@github-actions github-actions bot added the orange [process] Done by the Azul team label Feb 10, 2024
@dsotirho-ucsc dsotirho-ucsc added reqs [process] PR includes commit requiring ``make requirements`` upgrade [process] PR includes commit requiring manual upgrade labels Feb 10, 2024
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch 4 times, most recently from b810ca2 to c09e7d4 Compare February 13, 2024 17:17
Copy link

codecov bot commented Feb 13, 2024

Codecov Report

Attention: Patch coverage is 82.35294% with 3 lines in your changes missing coverage. Please review.

Project coverage is 85.34%. Comparing base (ccc785f) to head (70f573d).

Files with missing lines Patch % Lines
...llatlas/data/metadata/helpers/schema_validation.py 85.71% 2 Missing ⚠️
test/integration_test.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #5951      +/-   ##
===========================================
- Coverage    85.34%   85.34%   -0.01%     
===========================================
  Files          155      155              
  Lines        20785    20795      +10     
===========================================
+ Hits         17739    17747       +8     
- Misses        3046     3048       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Feb 13, 2024

Coverage Status

coverage: 85.36% (-0.003%) from 85.363%
when pulling 70f573d on issues/dsotirho-ucsc/5736-update-python-to-3-12
into ccc785f on develop.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from c09e7d4 to e4c1675 Compare February 14, 2024 22:45
@hannes-ucsc hannes-ucsc force-pushed the develop branch 3 times, most recently from 0762297 to 3f7f6ce Compare May 28, 2024 06:58
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from e4c1675 to 3ab9f81 Compare August 8, 2024 17:40
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from 3ab9f81 to ca1c534 Compare August 19, 2024 17:03
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch 2 times, most recently from c6c31f6 to 9dd4804 Compare August 27, 2024 18:25
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch 11 times, most recently from af1fdf3 to b2bb083 Compare September 20, 2024 19:59
nadove-ucsc
nadove-ucsc previously approved these changes Oct 2, 2024
@achave11-ucsc achave11-ucsc removed the upgrade [process] PR includes commit requiring manual upgrade label Oct 2, 2024
@achave11-ucsc achave11-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from c00b327 to cb9f617 Compare October 2, 2024 21:20
@achave11-ucsc achave11-ucsc added the sandbox [process] Resolution is being verified in sandbox deployment label Oct 2, 2024
@dsotirho-ucsc dsotirho-ucsc added the deploy:shared [process] PR requires deploying `shared` component label Oct 2, 2024
@achave11-ucsc achave11-ucsc removed their assignment Oct 2, 2024
@dsotirho-ucsc
Copy link
Contributor Author

IT failure due to combination of FIPS mode & the new bookworm-based Python image can be replicated with these commands:

$ sudo /usr/bin/docker run -it --rm --network gitlab-runner-net --env DOCKER_HOST=tcp://gitlab-dind:2375 --dns 172.21.0.2 --dns 169.254.169.253 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/library/docker:27.2.1 \
docker run -it --volume /var/run/docker.sock:/var/run/docker.sock docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/runner:latest \
docker run -it docker.io/library/python:3.12.6-slim-bookworm \
bash

# apt-get update && apt-get upgrade -y && apt-get -y install build-essential curl unzip

# install -m 0755 -d /etc/apt/keyrings

# curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc

# echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | tee /etc/apt/sources.list.d/docker.list

# apt-get update

Terminal log:

$ ssh gitlab.hcadev

[ec2-user@ip-172-21-0-99 ~]$ sudo /usr/bin/docker run -it --rm --network gitlab-runner-net --env DOCKER_HOST=tcp://gitlab-dind:2375 --dns 172.21.0.2 --dns 169.254.169.253 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/library/docker:27.2.1 \
> docker run -it --volume /var/run/docker.sock:/var/run/docker.sock docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/runner:latest \
> docker run -it docker.io/library/python:3.12.6-slim-bookworm \
> bash

root@02a62aa013df:/# apt-get update && apt-get upgrade -y && apt-get -y install build-essential curl unzip
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8787 kB]
Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [2468 B]
Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [188 kB]
Fetched 9232 kB in 2s (4113 kB/s)
Reading package lists... Done
Reading package lists... Done
...

root@02a62aa013df:/# install -m 0755 -d /etc/apt/keyrings

root@02a62aa013df:/# curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc

root@02a62aa013df:/# echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | tee /etc/apt/sources.list.d/docker.list
deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable

root@02a62aa013df:/# apt-get update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Get:4 https://download.docker.com/linux/debian bookworm InRelease [43.3 kB]
Get:5 https://download.docker.com/linux/debian bookworm/stable amd64 Packages [31.3 kB]
Fetched 74.6 kB in 1s (64.9 kB/s)
fatal error in libgcrypt, file ../../src/misc.c, line 92, function _gcry_fatal_error: requested algo not in md context

Fatal error: requested algo not in md context
Aborted

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch 2 times, most recently from 310fbe5 to ee38a8a Compare October 18, 2024 18:12
Resolves error with libgcrypt on bookworm:
Fatal error in libgcrypt, requested algo not in md context
Fulfills transitive requirement of distutils, which Python 3.12 removed from the standard library
For Python 3.12 support
Resolves error during `make requirements`:

  × Building wheel for jq (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [1390 lines of output]
      running bdist_wheel
      running build
      running build_ext
...
      jq.c:12124:55: error: no member named 'ob_digit' in 'struct _longobject'
                  const digit* digits = ((PyLongObject*)x)->ob_digit;
                                        ~~~~~~~~~~~~~~~~~~  ^
      jq.c:13608:47: error: no member named 'ob_digit' in 'struct _longobject'
          const digit* digits = ((PyLongObject*)b)->ob_digit;
                                ~~~~~~~~~~~~~~~~~~  ^
      41 warnings and 7 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for jq
Resolves errors originating from flake8 failing to parse f-strings under Python 3.12

E122 continuation line missing indentation or outdented
E131 continuation line unaligned for hanging indent
E202 whitespace before ')'
E202 whitespace before ']'
E202 whitespace before '}'
E203 whitespace before ':'
E221 multiple spaces before operator
E222 multiple spaces after operator
E225 missing whitespace around operator
E231 missing whitespace after ','
E231 missing whitespace after ':'
E231 missing whitespace after ';'
E271 multiple spaces after keyword
E272 multiple spaces before keyword
E702 multiple statements on one line (semicolon)
E713 test for membership should be 'not in'
W604 backticks are deprecated, use 'repr()'
Resolves error during `make requirements_update`:

19.11 Building wheels for collected packages: fastavro
19.11   Building wheel for fastavro (pyproject.toml): started
36.96   Building wheel for fastavro (pyproject.toml): finished with status 'error'
36.97   error: subprocess-exited-with-error
36.97
36.97   × Building wheel for fastavro (pyproject.toml) did not run successfully.
36.97   │ exit code: 1
36.97   ╰─> [73 lines of output]
36.97       <string>:31: SyntaxWarning: invalid escape sequence '\('
36.97       running bdist_wheel
36.97       running build
…
36.97       gcc: internal compiler error: Illegal instruction signal terminated program as
36.97       Please submit a full bug report,
36.97       with preprocessed source if appropriate.
36.97       See <file:///usr/share/doc/gcc-10/README.Bugs> for instructions.
36.97       error: command '/usr/bin/gcc' failed with exit code 4
36.97       [end of output]
Resolves error during test:

ERROR: test_openapi (integration_test.OpenAPIIntegrationTest.test_openapi) (component='service')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 1856, in test_openapi
    validate_spec(spec)
  File "/Users/daniel/repo/azul1/.venv/lib/python3.12/site-packages/openapi_spec_validator/shortcuts.py", line 17, in validate_spec
    return validator.validate(spec, spec_url=spec_url)
...
AttributeError: '_io.BufferedReader' object has no attribute 'decode'
Resolves DeprecationWarning during schema validation:

Traceback (most recent call last):
  File "/Users/daniel/.pyenv/versions/3.12.6/lib/python3.12/unittest/suite.py", line 306, in _tearDownPreviousClass
    tearDownClass()
AssertionError: ["{message : DeprecationWarning('Automatically retrieving remote references can be a security vulnerability and is discouraged by the JSON Schema specifications. Relying on this behavior is deprecated and will shortly become an error. If you are sure you want to remotely retrieve your reference and that it is safe to do so, you can find instructions for doing so via referencing.Registry in the referencing documentation (https://referencing.readthedocs.org).'), category : 'DeprecationWarning', filename : '/Users/daniel/repo/azul1/.venv/lib/python3.12/site-packages/jsonschema/exceptions.py', lineno : 475, line : None}", ...]
Resolves finding during `make pep8`:

F842 local variable 'cs' is annotated but never used
Resolves finding during `make pep8`

E122 continuation line missing indentation or outdented
Resolves error during test:

AttributeError: 'not_called' is not a valid assertion. Use a spec for the mock if 'not_called' is meant to be an attribute.
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from ee38a8a to 70f573d Compare October 30, 2024 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1 review [process] Lead requested changes once deploy:shared [process] PR requires deploying `shared` component orange [process] Done by the Azul team reqs [process] PR includes commit requiring ``make requirements`` sandbox [process] Resolution is being verified in sandbox deployment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants