Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance degradation moving from Python 3.10 to 3.12 #1304

Open
jkugler opened this issue Oct 1, 2024 · 1 comment
Open

Performance degradation moving from Python 3.10 to 3.12 #1304

jkugler opened this issue Oct 1, 2024 · 1 comment

Comments

@jkugler
Copy link

jkugler commented Oct 1, 2024 •
edited
Loading

I was running some timing today and noticed it was taking longer to run the schema validation under Python 3.12 than it was under Python 3.10. 16MB SBOM generated by Syft. About 45 seconds under 3.10, about 48 seconds under 3.12. Sadly, I'm not allowed to share the SBOM outside of my company. But, I can share my data. :)

Under profiling, the time differences are even more stark. 135 seconds under 3.10, 179 seconds under 3.12

Script:

import os
import sys

from cyclonedx.schema import SchemaVersion
from cyclonedx.validation.json import JsonStrictValidator

import cProfile
import pstats
from pstats import SortKey


def main():
    my_json_validator = JsonStrictValidator(SchemaVersion.V1_6)
    validation_errors = my_json_validator.validate_str(open('syft_sbom.json').read())
    if validation_errors:
        print('JSON invalid', 'ValidationError:', repr(validation_errors), sep='\n', file=sys.stderr)
        sys.exit(2)
    print('JSON valid')

v = sys.version_info
stat_file = os.path.expanduser(f"~/tmp/validation_{'_'.join([str(v.major), str(v.minor), str(v.micro)])}.data")
cProfile.run('main()', stat_file)
pstats.Stats(stat_file).sort_stats(SortKey.TIME, SortKey.CUMULATIVE).print_stats(50)
os.unlink(stat_file)

Versions installed:

Python 3.10

$ pip3 list
Package                   Version
------------------------- --------------
arrow                     1.3.0
attrs                     24.2.0
boolean.py                4.0
cyclonedx-python-lib      7.6.1
defusedxml                0.7.1
fqdn                      1.5.1
idna                      3.10
isoduration               20.11.0
jsonpointer               3.0.0
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
license-expression        30.3.1
packageurl-python         0.15.6
pip                       24.2
py-serializable           1.1.2
python-dateutil           2.9.0.post0
referencing               0.35.1
rfc3339-validator         0.1.4
rfc3987                   1.3.8
rpds-py                   0.20.0
setuptools                74.1.2
six                       1.16.0
sortedcontainers          2.4.0
types-python-dateutil     2.9.0.20240906
uri-template              1.3.0
webcolors                 24.8.0

Python 3.12. Identical, except for the lack of setuptools.

Package                   Version
------------------------- --------------
arrow                     1.3.0
attrs                     24.2.0
boolean.py                4.0
cyclonedx-python-lib      7.6.1
defusedxml                0.7.1
fqdn                      1.5.1
idna                      3.10
isoduration               20.11.0
jsonpointer               3.0.0
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
license-expression        30.3.1
packageurl-python         0.15.6
pip                       24.2
py-serializable           1.1.2
python-dateutil           2.9.0.post0
referencing               0.35.1
rfc3339-validator         0.1.4
rfc3987                   1.3.8
rpds-py                   0.20.0
six                       1.16.0
sortedcontainers          2.4.0
types-python-dateutil     2.9.0.20240906
uri-template              1.3.0
webcolors                 24.8.0

Timing under Python 3.10

Tue Oct  1 14:28:45 2024    /Users/tek30584/tmp/validation_3_10_15.data

         1168063553 function calls (1117721469 primitive calls) in 135.435 seconds

   Ordered by: internal time, cumulative time
   List reduced from 228 to 50 due to restriction <50>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
105980718/56440480   28.444    0.000  123.782    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:127(equal)
 56195901   27.838    0.000   54.892    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:106(_mapping_equal)
333189549   23.089    0.000   47.820    0.000 {built-in method builtins.isinstance}
168862378   13.585    0.000   24.731    0.000 /opt/homebrew/Cellar/[email protected]/3.10.15/Frameworks/Python.framework/Versions/3.10/lib/python3.10/abc.py:117(__instancecheck__)
168862378   11.145    0.000   11.145    0.000 {built-in method _abc._abc_instancecheck}
 99080476   10.389    0.000   17.554    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:112(<genexpr>)
        2    5.509    2.755  129.263   64.631 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:156(uniq)
 49551487    4.192    0.000   19.470    0.000 {built-in method builtins.all}
112424250/112423479    3.625    0.000    3.625    0.000 {built-in method builtins.len}
 50463582    1.743    0.000    1.743    0.000 {method 'items' of 'dict' objects}
 540652/7    0.808    0.000  135.364   19.338 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:397(descend)
   541226    0.526    0.000    1.491    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:340(evolve)
   137336    0.325    0.000    0.922    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:249(pointer)
   916230    0.303    0.000    0.446    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_types.py:90(is_type)
   541227    0.293    0.000    0.650    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:283(__attrs_post_init__)
  1081273    0.230    0.000    0.313    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_legacy_keywords.py:9(ignore_ref_siblings)
   402084    0.204    0.000    0.693    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:282(type)
   541227    0.172    0.000    0.172    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:294(<listcomp>)
   676801    0.166    0.000    0.297    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:160(create_resource)
  3497847    0.160    0.000    0.160    0.000 {method 'get' of 'dict' objects}
   916230    0.149    0.000    0.595    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:453(is_type)
   541227    0.143    0.000    0.793    0.000 <attrs generated init jsonschema.validators.create.<locals>.Validator>:1(__init__)
 116122/1    0.138    0.000  135.365  135.365 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:290(properties)
   137920    0.136    0.000    1.481    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:643(lookup)
   676801    0.132    0.000    0.132    0.000 <attrs generated init referencing._core.Resource>:1(__init__)
   539463    0.125    0.000    0.175    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/jsonschema.py:53(_legacy_dollar_id)
   274672    0.109    0.000    0.245    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/jsonschema.py:338(maybe_in_subresource)
   539463    0.107    0.000    0.282    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:222(id)
  2311156    0.106    0.000    0.106    0.000 {built-in method builtins.getattr}
   539462    0.093    0.000    0.375    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:690(in_subresource)
   804166    0.093    0.000    0.368    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:285(<genexpr>)
   137923    0.089    0.000    0.208    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/attr/_funcs.py:397(evolve)
137920/10603    0.066    0.000    6.044    0.001 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:274(ref)
   541226    0.066    0.000    0.071    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:1335(validator_for)
   584178    0.066    0.000    0.092    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_types.py:48(is_object)
   137920    0.062    0.000    0.119    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:405(get_or_retrieve)
   117312    0.061    0.000    0.295    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:36(additionalProperties)
   117312    0.060    0.000    0.085    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:69(find_additional_properties)
   402083    0.057    0.000    0.072    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:95(ensure_list)
   137920    0.056    0.000    0.266    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:706(_evolve)
        4    0.055    0.014    0.055    0.014 /opt/homebrew/Cellar/[email protected]/3.10.15/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py:343(raw_decode)
   137923    0.055    0.000    0.084    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/attr/_make.py:1699(fields)
   137920    0.054    0.000    1.536    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:460(_validate_reference)
   402083    0.048    0.000    0.393    0.000 {built-in method builtins.any}
  12449/2    0.040    0.000    6.101    3.051 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_legacy_keywords.py:124(items_draft6_draft7_draft201909)
   137921    0.036    0.000    0.036    0.000 <attrs generated init referencing._core.Resolver>:1(__init__)
   294105    0.035    0.000    0.046    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_types.py:52(is_string)
   563360    0.035    0.000    0.035    0.000 {method 'replace' of 'str' objects}
   117310    0.032    0.000    0.106    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:304(required)
   137920    0.029    0.000    0.029    0.000 <attrs generated init referencing._core.Retrieved>:1(__init__)

Timing under 3.12

Tue Oct  1 14:33:15 2024    /Users/tek30584/tmp/validation_3_12_6.data

         1167490383 function calls (1117148299 primitive calls) in 179.302 seconds

   Ordered by: internal time, cumulative time
   List reduced from 228 to 50 due to restriction <50>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 56195901   37.214    0.000   75.522    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:106(_mapping_equal)
333186659   34.868    0.000   66.689    0.000 {built-in method builtins.isinstance}
105980718/56440480   33.197    0.000  166.354    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:127(equal)
168862378   16.584    0.000   31.821    0.000 <frozen abc>:117(__instancecheck__)
168862378   15.237    0.000   15.237    0.000 {built-in method _abc._abc_instancecheck}
 99080476   13.222    0.000   21.989    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:112(<genexpr>)
 49551487    6.067    0.000   23.560    0.000 {built-in method builtins.all}
112424251/112423480    5.896    0.000    5.897    0.000 {built-in method builtins.len}
        2    5.838    2.919  172.158   86.079 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:156(uniq)
 50463582    4.473    0.000    4.473    0.000 {method 'items' of 'dict' objects}
 540652/7    0.900    0.000  179.232   25.605 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:397(descend)
   541226    0.597    0.000    1.556    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:340(evolve)
   137336    0.364    0.000    1.120    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:249(pointer)
  3495511    0.348    0.000    0.348    0.000 {method 'get' of 'dict' objects}
   916230    0.329    0.000    0.509    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_types.py:90(is_type)
   541227    0.307    0.000    0.588    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:283(__attrs_post_init__)
   402084    0.279    0.000    0.855    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:282(type)
  1081273    0.273    0.000    0.457    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_legacy_keywords.py:9(ignore_ref_siblings)
   676801    0.218    0.000    0.406    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:160(create_resource)
  2311156    0.206    0.000    0.206    0.000 {built-in method builtins.getattr}
   676801    0.187    0.000    0.187    0.000 <attrs generated init referencing._core.Resource>:1(__init__)
   916230    0.170    0.000    0.679    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:453(is_type)
 116122/1    0.141    0.000  179.232  179.232 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:290(properties)
   539463    0.132    0.000    0.213    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/jsonschema.py:53(_legacy_dollar_id)
   137920    0.131    0.000    1.733    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:643(lookup)
   804166    0.121    0.000    0.422    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:285(<genexpr>)
   539463    0.113    0.000    0.325    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:222(id)
   541227    0.111    0.000    0.699    0.000 <attrs generated init jsonschema.validators.create.<locals>.Validator>:1(__init__)
   274672    0.102    0.000    0.262    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/jsonschema.py:338(maybe_in_subresource)
   539462    0.094    0.000    0.419    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:690(in_subresource)
   137923    0.088    0.000    0.222    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/attr/_funcs.py:397(evolve)
   584178    0.078    0.000    0.117    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_types.py:48(is_object)
   117312    0.069    0.000    0.360    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:36(additionalProperties)
137920/10603    0.068    0.000    7.010    0.001 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:274(ref)
   137920    0.066    0.000    0.145    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:405(get_or_retrieve)
   117312    0.065    0.000    0.113    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:69(find_additional_properties)
   402083    0.065    0.000    0.089    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:95(ensure_list)
   402083    0.065    0.000    0.439    0.000 {built-in method builtins.any}
   549356    0.065    0.000    0.065    0.000 {method 'replace' of 'str' objects}
   137920    0.063    0.000    0.287    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:706(_evolve)
   541226    0.057    0.000    0.060    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:1335(validator_for)
        4    0.056    0.014    0.056    0.014 /opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/json/decoder.py:343(raw_decode)
   541226    0.055    0.000    0.055    0.000 {method 'setdefault' of 'dict' objects}
   137923    0.054    0.000    0.087    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/attr/_make.py:1699(fields)
   137920    0.048    0.000    1.782    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:460(_validate_reference)
   137921    0.047    0.000    0.047    0.000 <attrs generated init referencing._core.Resolver>:1(__init__)
  12449/2    0.042    0.000    7.073    3.537 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_legacy_keywords.py:124(items_draft6_draft7_draft201909)
   275845    0.041    0.000    0.041    0.000 {method 'startswith' of 'str' objects}
   137920    0.040    0.000    0.040    0.000 <attrs generated init referencing._core.Retrieved>:1(__init__)
   137920    0.039    0.000    0.039    0.000 {method 'get' of 'rpds.HashTrieMap' objects}
@jkugler
Copy link
Author

jkugler commented Oct 1, 2024

validation_data.TGZ

pStats run data from Python 3.10 and Python 3.12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jkugler