Poetry and dependency management — Python: Setting up the environment

• Adding and removing dependencies
• Dependency groups
• Updating dependencies
• Lock file

In this lesson, we'll take a closer look at working with dependencies with Poetry.

Adding and removing dependencies

In the last lesson, we created a project named hello. Let's return to it and add the colorama package — a dependency for hello. It is a popular library that allows you to colorize text in the terminal.

We will add dependencies with the command poetry add NAME:

poetry add colorama

Using version ^0.4.5 for colorama

Updating dependencies
Resolving dependencies... (0.8s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing colorama (0.4.5)

Now we will look at the tool.poetry.dependencies section of the pyproject.toml file. You'll find the following:

[tool.poetry.dependencies]
python = "^3.10"
colorama = "^0.4.5"

As you can see, colorama is installed in the virtual environment and appears in the list of project dependencies. If we want to run a code, we can run the poetry install command and get all the necessary dependencies.

Note the text "^0.4.5". It doesn't just mean specifically 0.4.5. The versions from 0.4.5 up to 0.5.0 will be considered compatible. You can also give a specific version, list several permissible versions, or manually specify a range. You can learn more about this variety in the documentation.

Now let's try to remove the dependencies with the command poetry remove NAME. Python will remove the uninstalled packages from the pyproject.toml file automatically:

poetry remove colorama

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing colorama (0.4.5)

Dependency groups

Many tools for developing Python projects are in Python. However, they're usually not needed to run the code. Take, for example, Pytest, a popular framework for writing tests. This package has many dependencies of its own. In a large project, these development tools can take up a lot of space.

The problem is that users do not need all these dependencies because they do not run any tests. However, the user gets all the dependencies with the program and wastes space by having to store them.

Fortunately, Poetry allows you to describe these packages as a group of dependencies. We will consider all created groups when building the virtual environment, but only the dependencies from the [tool.poetry.dependencies] group will reach the end user.

Add the Pytest package to the hello project. We'll put it in the dev group. Dev is the usual name for the development tools and environment. Let's see how it works:

poetry add --group dev pytest

Using version ^7.1.3 for pytest

Updating dependencies
Resolving dependencies... (0.8s)

Writing lock file
...

 • Installing pytest (7.1.3)

When you run the command yourself, pay attention to how many packages we will install with the pytest we need. There is a new entry in the pyproject.toml file:

[tool.poetry.group.dev.dependencies]
pytest = "^7.1.3"

As a result, we can run Pytest. Don't forget to write poetry run pytest instead of pytest. We install the program in a virtual environment and cannot see it from the outside. Let's try running it:

poetry run pytest

# Pytest hasn't run the tests yet
# because we haven't written them yet

====== test session starts ========
platform linux -- Python 3.10.4, ...
rootdir: /.../first
collected 0 items

====== no tests ran in 0.00s ======

The poetry run command doesn't just run commands from the virtual environment. It runs any program in the context of the virtual environment. For example, you can use the system program which to find out where the executable file pytest is:

poetry run which pytest

/../hello/.venv/bin/pytest

Now let's try to remove the dependency. To do this, you must indicate its group:

poetry remove --group dev pytest
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file
...
  • Removing pytest (7.1.3)

Updating dependencies

To update all dependencies, run the command poetry update. To update a specific dependency, run poetry update NAME. Dependencies will update according to the ranges specified.

Lock file

In the previous step, we created each new installation of dependencies and then updated it in the poetry.lock file. Let's discuss this file in more detail. As we already discussed, the file pyproject.toml specifies the dependencies. Each dependency may have its dependencies, which may have their dependencies etc. Dependencies of dependencies are transitive dependencies, and they're not always easy to deal with. The dependency system can be very confusing.

There is a term for such a situation — dependency hell. The problem is that we don't capture versions of transitive dependencies in any way. Here's an example:

• Our project has a dependent package, A, with a fixed version 1.3.2
• Dependency A has dependent package B with version *

In this situation, even without the lock file, the poetry install command would install:

• For A — the specified version
• For B — the latest available version from the repository

In other words, the choice of version isn't determined. If the author updates B and breaks backward compatibility, A will stop working, so the whole project will crash.

You can manually track dependencies of all dependencies and explicitly write their versions in pyproject.toml. But this method may not work because packages are constantly updated and changed. It's also hard to track manually because there are too many dependencies - even a project with five dependencies will have hundreds of transitive dependencies.

Another option would be to require that the creators of all libraries always specify the version. This option will not work because of the human factor this time.

There is one solution that will work, a lock file. It is essentially automated dependency tracking. The contents of a lock file look something like this:

[[package]]
name = "colorama"
version = "0.4.5"
description = "Cross-platform colored terminal text."
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"

[metadata]
lock-version = "1.1"
python-versions = "^3.10"
content-hash = "a07e5731d39f80ad23e37e53ffb7d54de3b1534e6306267399f3b25fcce783ad"

[metadata.files]
colorama = [
    {file = "colorama-0.4.5-py2.py3-none-any.whl", hash = "sha256:854bf444933e37f5824ae7bfc1e98d5bce2ebe4160d46b5edf346a89358e99da"},
    {file = "colorama-0.4.5.tar.gz", hash = "sha256:e6c6b4334fc50988a639d9b98aa429a0b57da6e17b9a44f0451f930b6967b7a4"},
]
...

The first run of the dependency installer will create this file. All installed dependencies are written there, including transitive ones with versions and hash sums.

On subsequent runs, the `poetry install' command will always put the data from the lock file. It works even if you delete the .venv folder or add new versions of packages to pyproject.toml. Running it after any amount of time will produce the same result. Now we can say that the project will run for any user at any time.

The lock file doesn't affect the behavior of the `update' command for direct dependencies. Let's say the package in our pyproject.toml file has a new version. In this case, Poetry will check if it's possible to update. If so, it will download the new version and automatically update the lock file.

If we don't want to install packages, we can use the poetry update --lock command. It will check which the new versions match the versions specified in the configuration and only update the lock file.