Dependency Management with Poetry | by Herman Michaels

By Jessie Hobb On Feb 1, 2023

Using pyenv and poetry to organize Python

One non-trivial thing about Python is managing its versions, environments, dependencies — and thus shipping projects to other people. In this post we will have a look at the packaging and dependency manager poetry — which — in the author’s humble opinion — is the best way of doing said things.

Before getting started, let us define some terms and clear up a bit of confusion, which also I have fallen victim to initially: in addition to poetry, we will use pyenv — a Python installation manager. pyenv lets us manage different Python versions on our system, s.t. we can for example use Python 2.7 for project A, and 3.10 for project B — in addition preventing a pollution of our Python installation, if we were to simply install all needed packages to our system Python via pip install ... .

On top of this, there are different packaging and dependency managers — such as poetry, but for example also pipenv, venv, or conda. In this post we will highlight poetry, which — as I mentioned earlier — I personally find the best of these solutions.

We first install pyenv (assuming you are on a Linux machine, for all others operating systems please follow the installation guide):

curl https://pyenv.run | bash

Then, in pyenv, we install the Python version we want to use for our project, e.g.:

pyenv install -v 3.10.0

With Python installed as described, we switch our focus to poetry.

Install via:

curl -sSL https://install.python-poetry.org | python3 -

Then, change into your Python project folder, and point poetry to the Python version you want to use — in our example case this is done via (you might need a pyproject.toml file, see below):

pyenv local 3.10.0poetry env use $(which python)

Poetry functions using two files: pyproject.toml and poetry.lock.

In the pyproject.toml, we describe needed dependencies / packages, for example torch. The developer, after adding all needed dependencies, then runs poetry update.

This triggers the creation of the poetry.lock file, which “captures” and describes the exact environment consisting of all packets from the pyproject.toml. When now a user or another developer downloads the repo / code, they simply run poetry install, and this will install all dependencies — resulting in them obtaining exactly the same Python environment as intended by the first developer.

This is a huge advantage, and a must — as opposed to everybody just installing their own dependencies — not doing this just causes too much trouble.

To use your environment (run scripts, …), run poetry shell. This activates your environment, and in it you can work as usual — e.g. run Python programs via python file.py.

To conclude, let us put this all together in a sample project — and for the first time also unveil contents of a sample pyproject.toml file.

Our project will consist of three files: main.py, pyproject.toml, and the auto-generated poetry.lock file.

main.py has the following content:

import matplotlib.pyplot as plt
import numpy as npdef plot():
x = np.linspace(0, 10, 50)
y = np.sin(x)
plt.plot(x, y)
plt.savefig("plot.png")
if __name__ == "__main__":
plot()

As we can see, we use numpy and matplotlib to generate a sinus curve and plot it. These modules thus need to be installed, which — ofc — we do with poetry.

Let us take a look at the contents of the pyproject.toml:

The first section contains some metadata about the project:

[tool.poetry]
name = "myproject"
version = "0.1.0"
description = "…"
authors = ["hermanmichaels <[email protected]>"]

After this, it is time to define needed dependencies. You can (which is advised) also define a specific version for each package, to make the poetry update process deterministic. Further, here we also define which Python version is expected. The following snippet installs matplotlib andnumpy, as well as some other packets I like to use for convenience (more on that in a future post!):

[tool.poetry.dependencies]
python = "3.10"
matplotlib = "3.5.1"
mypy = "0.910"
numpy = "1.22.3"
black = "22.3.0"

Putting it together:

[tool.poetry]
name = "myproject"
version = "0.1.0"
description = "…"
authors = ["hermanmichaels <[email protected]>"][tool.poetry.dependencies]
python = "3.10"
matplotlib = "3.5.1"
mypy = "0.910"
numpy = "1.22.3"
black = "22.3.0"

As explained above, to generate the poetry.lock file, we run poetry update. Then, another person using this project, can simply run poetry install to obtain all our dependencies. You or them can then execute the programm by running:

poetry shell

python main.py

Let’s wrap up with a quick summary: poetry is a (the best, in my opinion) packaging and dependency manager. It builds upon a working Python installation, which I recommend to manage via the Python installation manager pyenv.

After following the installation and setup steps above, create a new project, and add a pyproject.toml file. In this, define all needed dependencies.

Then, run:

poetry update to generate the lock file (or update it, when having added new dependencies), which will be used by others to obtain the same set of packets
poetry install to install all packets from the lock file (e.g., when you downloaded a new poetry project and want to install the dependencies — or your colleagues pushed a new version containing new packets)
poetry shell to enter the environment and run any Python script

This wraps up this introduction to poetry. Hope you find this useful in your future work —feel free to come back for more!