Recommended flow for adding/removing files locally, pushing to server and updating on server

[codeananda]

• I work locally with poetry - fine.
• I spin up an EC2 instance, clone my repo and poetry install - fine.
• I make updates locally (poetry add package, poetry remove package, poetry add --group dev package etc.) and push pyproject.toml and poetry.lock to repo - fine.
• On EC2, I pull changes from repo and... ???

I thought poetry update or poetry lock were what I needed. After all, I have a new poetry.lock, so running poetry lock should update my env with the contents of the lock file. Stuff happened when I ran poetry update or poetry lock...

But it didn't work properly until I ran poetry install.

Does anyone else think this is odd? Do I need to run poetry install every time I add/remove something? What is the point of poetry lock or poetry update then?

[radoering]

On EC2, I pull changes from repo and... ???

poetry install

What is the point of poetry lock or poetry update then?

Isn't it clear from reading the docs?

poetry lock locks packages and poetry install installs packages. I think that perfectly makes sense. poetry update is not that self-explanatory about what's updated (spoiler: the lock file and the environment but not the pyproject.toml) but that's why there is documentation...

[codeananda]

Isn't it clear from reading the docs?

Unfortunately, no.

One reason for this confusion: if I manually add a library to pyproject.toml (assuming I've already run poetry install earlier), PyCharm displays a popup saying

poetry.lock is out of date
Run poetry lock, poetry lock --no-update or poetry update

So, I run poetry lock/update to 'update' my environment and add the library to it.

Are you saying that actually I should run poetry install in this case instead of poetry lock/update?

(Fully aware this could be a PyCharm issue and not a docs issue)

poetry lock locks packages and poetry install installs packages. I think that perfectly makes sense.

When you say it like that, it does.

So then I ask: why do we need poetry lock? I thought poetry automatically locked packages to their specific versions? That seems to be a key selling point.

[radoering]

if I manually add a library to pyproject.toml

Are you saying that actually I should run poetry install in this case instead of poetry lock/update?

If you manually change pyproject.toml, you have to lock before running poetry install. poetry install is only the right choice if your lock file is consistent with your pyproject.toml. poetry lock only changes the lock file, but does not install. (poetry lock --no-update only does necessary changes for manually added/removed dependencies and keeps the rest.) poetry update updates all dependencies in the lock file according to the constraints in your pyproject.toml and installs them.

So then I ask: why do we need poetry lock?

You need poetry lock if you want to update the lock file without installing anything after editing the dependencies in the pyproject.toml manually. If you always use poetry add and poetry remove, you normally do not need poetry lock.

There are different workflows and you do not need all commands in all workflows:

Workflow 1:

• manually add/remove dependencies in pyproject.toml
• use poetry lock and poetry install
• no need for poetry add and poetry remove
• almost no need for poetry update (poetry lock does almost the same without installing)

Workflow 2:

• use poetry add and poetry remove
• do not edit dependencies in pyproject.toml manually
• use poetry install in CI
• use poetry update to update locked dependencies
• no need for poetry lock

There are further workflows and variants for sure...

[codeananda]

Thank you for your detailed explanation! I really appreciate it.

Maybe I missed it in the docs, but it doesn't seem like these flows are made clear anywhere.

poetry install is only the right choice if your lock file is consistent with your pyproject.toml

This is the exact kind of wording that makes poetry/env management confusing.

If my lock file is consistent with my pyproject.toml, why would I need to install anything? Consistent means they are the same, right?

Do you mean pyproject.toml and poetry.lock are consistent (the same) but then I manually add a brand new dependency to pyproject.toml? In the instance I add it, the two files are now inconsistent and thus I need to run poetry install? Actually, no you can't mean that because you said I need to lock first...

[radoering]

Maybe I missed it in the docs, but it doesn't seem like these flows are made clear anywhere.

No, I do not think they are and I do not know if it makes sense to emphasize these workflows. I assume these are not the only possible workflows. There are so many subtly different use cases...

If my lock file is consistent with my pyproject.toml, why would I need to install anything?

There are three "things" that can be consistent or not:

• pyproject.toml
• poetry.lock
• the environment where you install the packages

pyproject.toml and poetry.lock can be consistent without an environment. That is completely independent of which packages are installed. You have to install to make your environment consistent with pyproject.toml and poetry.lock.

Consistent means they are the same, right?

Not exactly. pyproject.toml and poetry.lock will never be "the same". Usually, pyproject.toml only contains direct dependencies and allows ranges of versions. poetry.lock contains all (direct and transitive) dependencies and contains exact versions. Consistent means that the lock file contains a valid solution for the dependencies defined in the pyproject.toml.

but then I manually add a brand new dependency to pyproject.toml? In the instance I add it, the two files are now inconsistent

When you manually add the dependency (compared to poetry add), then yes.

and thus I need to run poetry install? Actually, no you can't mean that because you said I need to lock first...

Before installing, pyproject.toml and poetry.lock must be consistent. Typically, after manually adding a dependency to pyproject.toml, you run poetry lock --no-update to make the lock file consistent and afterwards run poetry install to update your environment. Alternatively, you can achieve all three actions by using poetry add, which adds the dependency to pyproject.toml, locks and installs it.

[codeananda]

I assume these are not the only possible workflows. There are so many subtly different use cases...

I'm sure that's the case. There's probably 80/20 distribution, though. A small percentage of the workflows likely make up a large percentage of the use cases. How to use poetry to manage your environment locally (and also when deploying/working on remote machines) seems like a pretty common use case to me. I certainly would have found it valuable!

I appreciate the time and effort you've put in to answering my questions!