Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Highly Variable and Degraded Performance from PyPi installation #110

Open
coltonbh opened this issue Apr 10, 2024 · 0 comments
Open

Highly Variable and Degraded Performance from PyPi installation #110

coltonbh opened this issue Apr 10, 2024 · 0 comments
Labels
unconfirmed This report has not yet been confirmed by the developers

Comments

@coltonbh
Copy link
Contributor

Describe the bug

Running calculations using xtb installed using pip leads to highly variable and often severely degraded performance (up to 3 orders of magnitude slower runtime). This performance is stochastic as calculations will sometimes take 3 orders of magnitude less time too!

To Reproduce

Create one conda environment for xtb:

conda create -n xtb xtb-python -c conda-forge 

Create a python environment for a pip install

python -m venv env
source ./env/bin/activate
pip install xtb

Run the following script in each environment to compare timings. Run the script multiple time and you will see the performance of the pip install vary wildly from >.5 seconds to <.003 seconds.

import timeit

import numpy as np
from xtb.interface import Calculator, Param
from xtb.libxtb import VERBOSITY_FULL


numbers = np.array([8, 1, 1])
positions = np.array(
    [
        [0.0, 0.0, 0.0],
        [0.524, 1.687, 0.480],
        [1.146, -0.450, -1.354],
    ]
)


def calculate_xtb():
    calc = Calculator(Param.GFN2xTB, numbers, positions)
    calc.set_verbosity(VERBOSITY_FULL)
    calc.set_output("logs.out")
    calc.singlepoint()


# Number of times you want each function to be executed
number_of_executions = 10

# Time calculate_xtb
xtb_time = timeit.timeit(
    "calculate_xtb()", globals=globals(), number=number_of_executions
)
print(
    f"Average execution time for calculate_xtb: {xtb_time / number_of_executions:.5f} seconds"
)
  1. What is your setup, which version are you running (current master is subject to change)
    Latest conda and pip installs.

Timings in the conda environment will be stable around 0.003 seconds (on my laptop). Timings for the pip environment will vary from ~0.003 to >0.6 seconds.

Expected behaviour

The performance is the same for xtb regardless how it is installed. The performance will not vary over 3 orders of magnitude WITHIN the same install (i.e., the pip install will perform the same as the conda install on some executions and will perform 1000x slower on others. It appears random).

Additional context

The fact that the performance CAN be the same suggests something very funny is going on.

@coltonbh coltonbh added the unconfirmed This report has not yet been confirmed by the developers label Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
unconfirmed This report has not yet been confirmed by the developers
Projects
None yet
Development

No branches or pull requests

1 participant