Modern Python Environments - dependency and workspace management

Once you get through the pain of setting up a Python environment for a single "hello world"-esque application, you'll need to go through an even more difficult process of figuring out how to manage multiple environments for multiple Python projects. Some of the projects could be new while others are stale piles of code from ten years ago. Fortunately, there a number of tools available to help make dependency and workspace management easier.

In this article, we'll review the available tools for dependency and workspace management in order to solve the following problems:

1. Installing and switching between different versions of Python on the same machine
2. Managing dependencies and virtual environments
3. Reproducing environments

The Complete Python Guide:

1. Modern Python Environments - dependency and workspace management (this article!)
2. Testing in Python
3. Modern Test-Driven Development in Python
4. Python Code Quality
5. Python Type Checking
6. Documenting Python Code and Projects
7. Python Project Workflow

Contents

Installing Python

While you can download and install Python from the official binaries or with your system's package manager, you should steer clear of those approaches unless you're lucky enough to be using the same version of Python for all your current and future projects. Since this is probably not the case, we recommend installing Python with pyenv.

pyenv is tool that simplifies installing and switching between different versions of Python on the same machine. It keeps the system version of Python intact, which is required for some operating systems to run properly, while still making it easy to switch Python versions based on a specific project's requirements.

Unfortunately, pyenv does not work on Windows outside the Windows Subsystem for Linux. Check out pyenv-win if this is the case for you.

Once installed, you can easily install a specific version of Python like so:

$ pyenv install 3.8.5
$ pyenv install 3.8.6
$ pyenv install 3.9.0
$ pyenv install 3.10.2

$ pyenv versions
* system
  3.8.5
  3.8.6
  3.9.0
  3.10.2

You can then set your global Python version like so:

$ pyenv global 3.8.6

$ pyenv versions
  system
  3.8.5
* 3.8.6 (set by /Users/michael/.pyenv/version)
  3.9.0
  3.10.2

$ python -V
Python 3.8.6

Keep in mind that this does not modify or affect Python at the system-level.

In a similar manner you can set the Python interpreter for the current folder:

$ pyenv local 3.10.2

$ pyenv versions
  system
  3.8.5
  3.8.6
  3.9.0
* 3.10.2 (set by /Users/michael/repos/testdriven/python-environments/.python-version)

$ python -V
Python 3.10.2

Now, every time you run Python inside that folder, version 3.10.2 will be used.

If you find that you can't switch between different versions of Python, you may need to re-configure your shell profile. Refer to the installation guide for details.

Managing Dependencies

In this section we'll look at several tools for managing dependencies as well as virtual environments.

venv + pip

venv and pip (package installer for python), which come pre-installed with most versions of Python, are the most popular tools for managing virtual environments and packages, respectively. They are fairly simple to use.

Virtual environments prevent dependency version conflicts. You can install different versions of the same dependency in different virtual environments.

You can create a new virtual environment called my_venv inside the current folder like so:

$ python -m venv my_venv

With the environment created, you still need to activate it by sourcing the activate script inside the virtual environment:

$ source my_venv/bin/activate
(my_venv)$

To deactivate run deactivate. Then, to re-activate, run source my_venv/bin/activate within the root project directory.

Running which python while the virtual environment is activated will return the path to the Python interpreter inside the virtual environment:

(my_venv)$ which python

/Users/michael/repos/testdriven/python-environments/my_venv/bin/python

You can install packages local to your project by running pip install <package-name> with the virtual environment activated:

(my_venv)$ python -m pip install requests

pip downloads the package from PyPI (Python Package Index) and then makes it available to the Python interpreter inside the virtual environment.

For environment reproducibility, you'll typically want to keep a list of required packages for the project inside a requirements.txt file. You can manually create the file and add them or use the pip freeze command to generate it:

(my_venv)$ python -m pip freeze > requirements.txt

(my_venv)$ cat requirements.txt
certifi==2021.10.8
charset-normalizer==2.0.12
idna==3.3
requests==2.27.1
urllib3==1.26.8

Want to grab just the top-level dependencies (e.g., requests==2.27.1)? Check out pip-chill.

While both venv and pip are easy to use, they are very primitive when compared to more modern tools like Poetry and Pipenv. venv and pip know nothing about the version of Python it's working with. You have to manage all dependencies and virtual environments by hand. You have to create and manage the requirements.txt file yourself. What's more, you'll have to manually separate development (pytest, black, isort, ...) and production (Flask, Django, FastAPI, ..) dependencies using a requirements-dev.txt file.

requirements-dev.txt:

# prod
-r requirements.txt

# dev
black==22.1.0
coverage==6.3.2
flake8==4.0.1
ipython==8.0.1
isort==5.10.1
pytest-django==4.5.2
pytest-cov==3.0.0
pytest-xdist==2.5.0
pytest-mock==3.7.0

requirements.txt:

Django==4.0.2
django-allauth==0.49.0
django-crispy-forms==1.14.0
django-rq==2.5.1
django-rq-email-backend==0.1.3
gunicorn==20.1.0
psycopg2-binary==2.9.3
redis==4.1.4
requests==2.27.1
rq==1.10.1
whitenoise==6.0.0

Poetry and Pipenv combine the functionality of venv and pip. They also make it easy to separate development and production dependencies as well as enable deterministic builds via a lock file. They work well with pyenv.

Lock files pin down (or lock) all dependency versions throughout the entire dependency tree.

Poetry

Poetry is arguably the most feature-rich dependency management tool for Python. It comes with a powerful CLI used for creating and managing Python projects. Once installed, to scaffold a new project run:

$ poetry new sample-project
$ cd sample-project

This will create the following files and folders:

sample-project
├── README.rst
├── pyproject.toml
├── sample_project
│   └── __init__.py
└── tests
    ├── __init__.py
    └── test_sample_project.py

Dependencies are managed inside the pyproject.toml file:

[tool.poetry]
name = "sample-project"
version = "0.1.0"
description = ""
authors = ["John Doe <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.10"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

For more on pyproject.toml, the new Python package configuration file that treats "every project as a package", check out the What the heck is pyproject.toml? article.

To add new a dependency, simply run:

$ poetry add [--dev] <package name>

The --dev flag indicates that the dependency is meant to be used in development mode only. Development dependencies are not installed by default.

For example:

$ poetry add flask

This downloads and installs Flask from PyPI inside the virtual environment managed by Poetry, adds it along with all sub-dependencies to the poetry.lock file, and automatically adds it (a top-level dependency) to pyproject.toml:

[tool.poetry.dependencies]
python = "^3.10"
Flask = "^2.0.3"

Take note of the version constraint: "^2.0.3".

To run a command inside the virtual environment, prefix the command with poetry run. For example, to run tests with pytest:

$ poetry run python -m pytest

poetry run <command> will run commands inside the virtual environment. It doesn't activate the virtual environment, though. To activate Poetry's virtual environment you need to run poetry shell. To deactivate it, you can simply run the exit command. Consequently, you can activate your virtual environment before working on the project and deactivate once you're done or you can use poetry run <command> throughout development.

Finally, Poetry works well with pyenv. Review Managing environments from the official docs for more on this.

Pipenv

Pipenv attempts to solve the same problems that Poetry does:

1. Managing dependencies and virtual environments
2. Reproducing environments

Once installed, to create a new project with Pipenv, run:

$ mkdir sample-project
$ cd sample-project
$ pipenv --python 3.10.2

This will create a new virtual environment and add a Pipfile to the project:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]

[requires]
python_version = "3.10"

A Pipfile works much like the pyproject.toml file does back in Poetry land.

You can install a new dependency like so:

$ pipenv install [--dev] <package name>

The --dev flag indicates that the dependency is meant to be used in development mode only. Development dependencies are not installed by default.

For example:

$ pipenv install flask

As with Poetry, Pipenv downloads and installs Flask inside the virtual environment, pins all sub-dependencies in the Pipfile.lock file, and adds the top-level dependency to Pipfile.

To run a script inside the virtual environment managed by Pipenv, you need to run it with the pipenv run command. For example, to run tests with pytest, run:

$ pipenv run python -m pytest

Like Poetry, pipenv run <command> will run commands from inside the virtual environment. To activate Pipenv's virtual environment you need to run pipenv shell. To deactivate it, you can run exit.

Pipenv works well with pyenv too. For example, when you want to create a virtual environment from a Python version that you don't have installed, it will ask if you'd like install it first with pyenv:

$ pipenv --python 3.7.5

Warning: Python 3.7.5 was not found on your system…
Would you like us to install CPython 3.7.5 with Pyenv? [Y/n]: Y

Recommendations

Which should I use?

1. venv and pip
2. Poetry
3. Pipenv

It's recommended to start with venv and pip. They are the easiest to work with. Familiarize yourself with them and figure out on your own what they are good at and where they are lacking.

Poetry or Pipenv?

Since they both address the same problems, it comes down to personal preference.

Notes:

1. Publishing to PyPI is much easier with Poetry, so if you're creating a Python package go with Poetry.
2. Both projects are fairly slow when it comes to dependency resolution, so if you're using Docker you may want to steer clear of them both.
3. From an open-source development perspective, Poetry moves faster and is arguably more responsive to user feedback.

Additional Tools

In addition to the above tools take a look at the following for help with installing and switching between different versions of Python on the same machine, managing dependencies and virtual environments, and reproducing environments:

1. Docker is a platform for building, deploying, and managing containerized applications. It's perfect for creating reproducible environments.
2. Conda, which is quite popular with the data science and machine learning communities, can help with managing dependencies and virtual environments as well as reproducing environments.
3. When you just need to simplify switching between virtual environments and manage them in one place virtualenvwrapper and pyenv-virtualenv, a pyenv plugin, are worth looking at.
4. pip-tools simplifies dependency management and environment reproducibility. It's often coupled wth venv.

Managing a Project

Let's take a look on how to manage a Flask project using pyenv and Poetry.

First, create a new directory called "flask_example" and move inside it:

$ mkdir flask_example
$ cd flask_example

Second, set the Python version for the project with pyenv:

$ pyenv local 3.10.2

Next, initialize a new Python project with Poetry:

$ poetry init

Package name [flask_example]:
Version [0.1.0]:
Description []:
Author [Your name <[email protected]>, n to skip]:
License []:
Compatible Python versions [^3.10]:

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Do you confirm generation? (yes/no) [yes]

Add Flask:

$ poetry add flask

Last but not least, add pytest as a development dependency:

$ poetry add --dev pytest

Now that we have a basic environment set up, we can write a test for a single endpoint.

Add a file called test_app.py:

import pytest

from app import app


@pytest.fixture
def client():
    app.config['TESTING'] = True

    with app.test_client() as client:
        yield client


def test_health_check(client):
    response = client.get('/health-check/')

    assert response.status_code == 200

After that, add a basic Flask app to a new file called app.py:

from flask import Flask

app = Flask(__name__)


@app.route('/health-check/')
def health_check():
    return 'OK'


if __name__ == '__main__':
    app.run()

Now, to run the tests, run:

$ poetry run python -m pytest

And you can run the development server like so:

$ poetry run python -m flask run

The poetry run command runs a command inside Poetry's virtual environment.

Conclusion

This article looked at the most popular tools for addressing the following problems with regard to dependency and workspace management:

1. Installing and switching between different versions of Python on the same machine
2. Managing dependencies and virtual environments
3. Reproducing environments

It matters less about the specific tools you use in your workflow and more that you are able to resolve those problems. Pick and choose the tools that make it easy for you to develop in Python. Experiment. They exist to make your everyday development workflows easier so you can become as productive as you can be. Try all of them and use the ones that work with your style of development. No judgments.

Happy coding.

The Complete Python Guide:

1. Modern Python Environments - dependency and workspace management (this article!)
2. Testing in Python
3. Modern Test-Driven Development in Python
4. Python Code Quality
5. Python Type Checking
6. Documenting Python Code and Projects
7. Python Project Workflow