Update to Python 3.12.x (#5736) #5951

dsotirho-ucsc · 2024-02-10T01:59:25Z

Connected issues: #5736

Checklist

Author

PR is a draft
Target branch is develop
Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
On ZenHub, PR is connected to all issues it (partially) resolves
PR description links to connected issues
PR title matches¹ that of a connected issue _{or comment in PR explains why they're different}
PR title references all connected issues
For each connected issue, there is at least one commit whose title references that issue

¹ when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

Added p tag to titles of partial commits
This PR is labeled partial _{or completely resolves all connected issues}
This PR partially resolves each of the connected issues _{or does not have the partial label}

Author (chains)

This PR is blocked by previous PR in the chain _{or is not chained to another PR}
The blocking PR is labeled base _{or this PR is not chained to another PR}
This PR is labeled chained _{or is not chained to another PR}

Author (reindex, API changes)

Added r tag to commit title _{or the changes introduced by this PR will not require reindexing of any deployment}
This PR is labeled reindex:dev _{or the changes introduced by it will not require reindexing of dev}
This PR is labeled reindex:anvildev _{or the changes introduced by it will not require reindexing of anvildev}
This PR is labeled reindex:anvilprod _{or the changes introduced by it will not require reindexing of anvilprod}
This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod _{or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod}
This PR and its connected issues are labeled API _{or this PR does not modify a REST API}
Added a (A) tag to commit title for backwards (in)compatible changes _{or this PR does not modify a REST API}
Updated REST API version number in app.py _{or this PR does not modify a REST API}

Author (upgrading deployments)

Ran make docker_images.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
Added u tag to commit title _{or this PR does not require upgrading deployments}
This PR is labeled upgrade _{or does not require upgrading deployments}
This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (hotfixes)

Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
Reverted the temporary hotfixes for any connected issues _{or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues connected to this PR}

Author (before every review)

Rebased PR branch on develop, squashed old fixups
Ran make requirements_update _{or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile}
Added R tag to commit title _{or this PR does not modify requirements*.txt}
This PR is labeled reqs _{or does not modify requirements*.txt}
make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}

Peer reviewer (after approval)

PR is not a draft
Ticket is in Review requested column
PR is awaiting requested review from system administrator
PR is assigned to only the system administrator

System administrator (after approval)

Actually approved the PR
Labeled connected issues as demo or no demo
Commented on connected issues about demo expectations _{or all connected issues are labeled no demo}
Decided if PR can be labeled no sandbox
A comment to this PR details the completed security design review
PR title is appropriate as title of merge commit
N reviews label is accurate
Moved connected issues to Approved column
PR is assigned to only the operator

Operator (before pushing merge the commit)

System administrator

Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
PR is assigned to only the operator

Operator (before pushing merge the commit)

Operator (chain shortening)

Changed the target branch of the blocked PR to develop _{or this PR is not labeled base}
Removed the chained label from the blocked PR _{or this PR is not labeled base}
Removed the blocking relationship from the blocked PR _{or this PR is not labeled base}
Removed the base label from this PR _{or this PR is not labeled base}

Operator (after pushing the merge commit)

Operator (reindex)

Operator

Propagated the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels to the next promotion PRs _{or this PR carries none of these labels}
Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels, from the description of this PR to that of the next promotion PRs _{or this PR carries none of these labels}
PR is assigned to no one

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

codecov · 2024-02-13T17:39:52Z

Codecov Report

Attention: Patch coverage is 82.35294% with 3 lines in your changes missing coverage. Please review.

Project coverage is 85.34%. Comparing base (ccc785f) to head (70f573d).

Files with missing lines	Patch %	Lines
...llatlas/data/metadata/helpers/schema_validation.py	85.71%	2 Missing ⚠️
test/integration_test.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #5951      +/-   ##
===========================================
- Coverage    85.34%   85.34%   -0.01%     
===========================================
  Files          155      155              
  Lines        20785    20795      +10     
===========================================
+ Hits         17739    17747       +8     
- Misses        3046     3048       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coveralls · 2024-02-13T17:39:57Z

coverage: 85.36% (-0.003%) from 85.363%
when pulling 70f573d on issues/dsotirho-ucsc/5736-update-python-to-3-12
into ccc785f on develop.

dsotirho-ucsc · 2024-10-08T22:42:46Z

IT failure due to combination of FIPS mode & the new bookworm-based Python image can be replicated with these commands:

$ sudo /usr/bin/docker run -it --rm --network gitlab-runner-net --env DOCKER_HOST=tcp://gitlab-dind:2375 --dns 172.21.0.2 --dns 169.254.169.253 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/library/docker:27.2.1 \
docker run -it --volume /var/run/docker.sock:/var/run/docker.sock docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/runner:latest \
docker run -it docker.io/library/python:3.12.6-slim-bookworm \
bash

# apt-get update && apt-get upgrade -y && apt-get -y install build-essential curl unzip

# install -m 0755 -d /etc/apt/keyrings

# curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc

# echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | tee /etc/apt/sources.list.d/docker.list

# apt-get update

Terminal log:

$ ssh gitlab.hcadev

[ec2-user@ip-172-21-0-99 ~]$ sudo /usr/bin/docker run -it --rm --network gitlab-runner-net --env DOCKER_HOST=tcp://gitlab-dind:2375 --dns 172.21.0.2 --dns 169.254.169.253 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/library/docker:27.2.1 \
> docker run -it --volume /var/run/docker.sock:/var/run/docker.sock docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/runner:latest \
> docker run -it docker.io/library/python:3.12.6-slim-bookworm \
> bash

root@02a62aa013df:/# apt-get update && apt-get upgrade -y && apt-get -y install build-essential curl unzip
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8787 kB]
Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [2468 B]
Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [188 kB]
Fetched 9232 kB in 2s (4113 kB/s)
Reading package lists... Done
Reading package lists... Done
...

root@02a62aa013df:/# install -m 0755 -d /etc/apt/keyrings

root@02a62aa013df:/# curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc

root@02a62aa013df:/# echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | tee /etc/apt/sources.list.d/docker.list
deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable

root@02a62aa013df:/# apt-get update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Get:4 https://download.docker.com/linux/debian bookworm InRelease [43.3 kB]
Get:5 https://download.docker.com/linux/debian bookworm/stable amd64 Packages [31.3 kB]
Fetched 74.6 kB in 1s (64.9 kB/s)
fatal error in libgcrypt, file ../../src/misc.c, line 92, function _gcry_fatal_error: requested algo not in md context

Fatal error: requested algo not in md context
Aborted

Resolves error with libgcrypt on bookworm: Fatal error in libgcrypt, requested algo not in md context

Fulfills transitive requirement of distutils, which Python 3.12 removed from the standard library

For Python 3.12 support

Resolves error during `make requirements`: × Building wheel for jq (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [1390 lines of output] running bdist_wheel running build running build_ext ... jq.c:12124:55: error: no member named 'ob_digit' in 'struct _longobject' const digit* digits = ((PyLongObject*)x)->ob_digit; ~~~~~~~~~~~~~~~~~~ ^ jq.c:13608:47: error: no member named 'ob_digit' in 'struct _longobject' const digit* digits = ((PyLongObject*)b)->ob_digit; ~~~~~~~~~~~~~~~~~~ ^ 41 warnings and 7 errors generated. error: command '/usr/bin/clang' failed with exit code 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for jq

Resolves errors originating from flake8 failing to parse f-strings under Python 3.12 E122 continuation line missing indentation or outdented E131 continuation line unaligned for hanging indent E202 whitespace before ')' E202 whitespace before ']' E202 whitespace before '}' E203 whitespace before ':' E221 multiple spaces before operator E222 multiple spaces after operator E225 missing whitespace around operator E231 missing whitespace after ',' E231 missing whitespace after ':' E231 missing whitespace after ';' E271 multiple spaces after keyword E272 multiple spaces before keyword E702 multiple statements on one line (semicolon) E713 test for membership should be 'not in' W604 backticks are deprecated, use 'repr()'

Resolves error during `make requirements_update`: 19.11 Building wheels for collected packages: fastavro 19.11 Building wheel for fastavro (pyproject.toml): started 36.96 Building wheel for fastavro (pyproject.toml): finished with status 'error' 36.97 error: subprocess-exited-with-error 36.97 36.97 × Building wheel for fastavro (pyproject.toml) did not run successfully. 36.97 │ exit code: 1 36.97 ╰─> [73 lines of output] 36.97 <string>:31: SyntaxWarning: invalid escape sequence '\(' 36.97 running bdist_wheel 36.97 running build … 36.97 gcc: internal compiler error: Illegal instruction signal terminated program as 36.97 Please submit a full bug report, 36.97 with preprocessed source if appropriate. 36.97 See <file:///usr/share/doc/gcc-10/README.Bugs> for instructions. 36.97 error: command '/usr/bin/gcc' failed with exit code 4 36.97 [end of output]

Resolves error during test: ERROR: test_openapi (integration_test.OpenAPIIntegrationTest.test_openapi) (component='service') ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/daniel/repo/azul1/test/integration_test.py", line 1856, in test_openapi validate_spec(spec) File "/Users/daniel/repo/azul1/.venv/lib/python3.12/site-packages/openapi_spec_validator/shortcuts.py", line 17, in validate_spec return validator.validate(spec, spec_url=spec_url) ... AttributeError: '_io.BufferedReader' object has no attribute 'decode'

Resolves DeprecationWarning during schema validation: Traceback (most recent call last): File "/Users/daniel/.pyenv/versions/3.12.6/lib/python3.12/unittest/suite.py", line 306, in _tearDownPreviousClass tearDownClass() AssertionError: ["{message : DeprecationWarning('Automatically retrieving remote references can be a security vulnerability and is discouraged by the JSON Schema specifications. Relying on this behavior is deprecated and will shortly become an error. If you are sure you want to remotely retrieve your reference and that it is safe to do so, you can find instructions for doing so via referencing.Registry in the referencing documentation (https://referencing.readthedocs.org).'), category : 'DeprecationWarning', filename : '/Users/daniel/repo/azul1/.venv/lib/python3.12/site-packages/jsonschema/exceptions.py', lineno : 475, line : None}", ...]

Resolves finding during `make pep8`: F842 local variable 'cs' is annotated but never used

Resolves finding during `make pep8` E122 continuation line missing indentation or outdented

Resolves error during test: AttributeError: 'not_called' is not a valid assertion. Use a spec for the mock if 'not_called' is meant to be an attribute.

github-actions bot added the orange [process] Done by the Azul team label Feb 10, 2024

dsotirho-ucsc added reqs [process] PR includes commit requiring ``make requirements`` upgrade [process] PR includes commit requiring manual upgrade labels Feb 10, 2024

dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch 4 times, most recently from b810ca2 to c09e7d4 Compare February 13, 2024 17:17

dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from c09e7d4 to e4c1675 Compare February 14, 2024 22:45

achave11-ucsc force-pushed the develop branch from ed4b5e1 to 858f6fb Compare February 15, 2024 21:38

hannes-ucsc force-pushed the develop branch 3 times, most recently from 0762297 to 3f7f6ce Compare May 28, 2024 06:58

dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from e4c1675 to 3ab9f81 Compare August 8, 2024 17:40

dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from 3ab9f81 to ca1c534 Compare August 19, 2024 17:03

hannes-ucsc force-pushed the develop branch from f54b36e to d769f27 Compare August 20, 2024 22:09

dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch 2 times, most recently from c6c31f6 to 9dd4804 Compare August 27, 2024 18:25

dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch 11 times, most recently from af1fdf3 to b2bb083 Compare September 20, 2024 19:59

dsotirho-ucsc assigned nadove-ucsc Oct 2, 2024

nadove-ucsc previously approved these changes Oct 2, 2024

View reviewed changes

nadove-ucsc assigned achave11-ucsc and unassigned nadove-ucsc Oct 2, 2024

achave11-ucsc removed the upgrade [process] PR includes commit requiring manual upgrade label Oct 2, 2024

achave11-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from c00b327 to cb9f617 Compare October 2, 2024 21:20

achave11-ucsc added the sandbox [process] Resolution is being verified in sandbox deployment label Oct 2, 2024

dsotirho-ucsc added the deploy:shared [process] PR requires deploying `shared` component label Oct 2, 2024

achave11-ucsc removed their assignment Oct 2, 2024

dsotirho-ucsc mentioned this pull request Oct 8, 2024

Update to Python 3.12.x #5736

Open

dsotirho-ucsc dismissed nadove-ucsc’s stale review via 310fbe5 October 14, 2024 17:03

dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch 2 times, most recently from 310fbe5 to ee38a8a Compare October 18, 2024 18:12

dsotirho-ucsc added 15 commits October 23, 2024 10:04

Update to Python 3.12.x (#5736)

965eea9

Disable FIPS mode in Docker container during apt-get update

66dda07

Resolves error with libgcrypt on bookworm: Fatal error in libgcrypt, requested algo not in md context

Make requirements_update download binaries, not source packages

3815247

[R] Add setuptools as a requirement

e7c8b9b

Fulfills transitive requirement of distutils, which Python 3.12 removed from the standard library

[R] Update azul-chalice to 1.31.2+13

8419baf

For Python 3.12 support

Remove unassigned type-hinted variable

610f387

Resolves finding during `make pep8`: F842 local variable 'cs' is annotated but never used

Reformat ternary operation

f6e773b

Resolves finding during `make pep8` E122 continuation line missing indentation or outdented

Fix broken assert of Mock object's not_called property

a4f2da1

Resolves error during test: AttributeError: 'not_called' is not a valid assertion. Use a spec for the mock if 'not_called' is meant to be an attribute.

Ignore DeprecationWarning from datetime, add FIXME (#5953)

8c0d34b

[R] Ran make requirements_update

70f573d

dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5736-update-python-to-3-12 branch from ee38a8a to 70f573d Compare October 30, 2024 00:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to Python 3.12.x (#5736) #5951

Update to Python 3.12.x (#5736) #5951

dsotirho-ucsc commented Feb 10, 2024 •

edited by achave11-ucsc

Loading

codecov bot commented Feb 13, 2024 •

edited

Loading

coveralls commented Feb 13, 2024 •

edited

Loading

dsotirho-ucsc commented Oct 8, 2024

Update to Python 3.12.x (#5736) #5951

Are you sure you want to change the base?

Update to Python 3.12.x (#5736) #5951

Conversation

dsotirho-ucsc commented Feb 10, 2024 • edited by achave11-ucsc Loading

Checklist

Author

Author (partiality)

Author (chains)

Author (reindex, API changes)

Author (upgrading deployments)

Author (hotfixes)

Author (before every review)

Peer reviewer (after approval)

System administrator (after approval)

Operator (before pushing merge the commit)

System administrator

Operator (before pushing merge the commit)

Operator (chain shortening)

Operator (after pushing the merge commit)

Operator (reindex)

Operator

Shorthand for review comments

codecov bot commented Feb 13, 2024 • edited Loading

Codecov Report

coveralls commented Feb 13, 2024 • edited Loading

dsotirho-ucsc commented Oct 8, 2024

dsotirho-ucsc commented Feb 10, 2024 •

edited by achave11-ucsc

Loading

codecov bot commented Feb 13, 2024 •

edited

Loading

coveralls commented Feb 13, 2024 •

edited

Loading