Python Project Workflow

Last updated January 5th, 2021

Thus far, in this series, we've covered:

  1. Modern Python Environments - dependency and workspace management
  2. Testing in Python
  3. Modern Test-Driven Development in Python
  4. Python Code Quality
  5. Python Type Checking
  6. Documenting Python Code and Projects

In this article, you'll glue everything together as you develop a single project from start to finish. After developing the basic project, you'll:

  1. Wire up CI/CD with GitHub Actions
  2. Configure coverage reporting with CodeCov
  3. Publish the package to PyPi and the docs to Read the Docs
  4. Update PyPI and Read the Docs via GitHub Actions

Feel free to swap out GitHub Actions for a similar CI/CD tool like GitLab CI, Bitbucket Pipelines, or CircleCI.

Contents

Project Setup

Let's build a random quote generator to return a randomly selected quote from set of quotes.

Initialize project

First, let's create a new folder for our project:

$ mkdir random-quote-generator
$ cd random-quote-generator

Initialize the project with Poetry:

$ poetry init

Package name [random_quote_generator]:
Version [0.1.0]:
Description []:
Author [Your name <[email protected]>, n to skip]:
License []:
Compatible Python versions [^3.9]:

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Do you confirm generation? (yes/no) [yes]

For more on Poetry, check out the Modern Python Environments - Dependency and workspace management article.

Your project name must be unique since you'll be uploading it to PyPI. So, to avoid name collisions add a unique string to the package name in pyproject.toml.

For example:

[tool.poetry]
name = "random-quote-generator-9308"
version = "0.1.0"
description = ""
authors = ["Michael Herman <[email protected]>"]

[tool.poetry.dependencies]
python = ">3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Create a new repository on GitHub:

GitHub new repository

Next, initialize a git repository inside your project:

$ git init
$ git add pyproject.toml
$ git commit -m "first commit"
$ git branch -M main
$ git remote add origin [email protected]:<your-github-username>/random-quote-generator.git
$ git branch --set-upstream-to=origin/main main
$ git pull origin main --rebase
$ git push -u origin main

With the basic setup complete, let's continue by adding the following development dependencies:

Review the Python Code Quality article for details on these dependencies.

Install:

$ poetry add --dev pytest pytest-cov black isort flake8 bandit safety

Add the new poetry.lock file as well as the updated pyproject.toml file to git:

$ git add poetry.lock pyproject.toml

Build the Project

After that, create a new folder called "random_quote_generator". Inside that folder, add an __init__.py file, so it's treated as a module, along with a quotes.py file.

random-quote-generator
├── poetry.lock
├── pyproject.toml
└── random_quote_generator
    ├── __init__.py
    └── quotes.py

Inside quotes.py, add:

quotes = [
    {
        "quote": "A long descriptive name is better than a short "
        "enigmatic name. A long descriptive name is better "
        "than a long descriptive comment.",
        "author": "Robert C. Martin",
    },
    {
        "quote": "You should name a variable using the same "
        "care with which you name a first-born child.",
        "author": "Robert C. Martin",
    },
    {
        "quote": "Any fool can write code that a computer "
        "can understand. Good programmers write code"
        " that humans can understand.",
        "author": "Martin Fowler",
    },
]

Nothing special there. Just a list of dictionaries, one for each quote. Next, create a new folder in the project root called "tests" and add the following files:

tests
├── __init__.py
└── test_get_quote.py

test_get_quote.py:

from random_quote_generator import get_quote
from random_quote_generator.quotes import quotes


def test_get_quote():
    """
    GIVEN
    WHEN get_quote is called
    THEN random quote from quotes is returned
    """

    quote = get_quote()

    assert quote in quotes

Run the test:

$ poetry run python -m pytest tests

It should fail:

E   ImportError: cannot import name 'get_quote' from 'random_quote_generator'

Next, add a new file to "random_quote_generator" called get_quote.py:

import random

from random_quote_generator.quotes import quotes


def get_quote() -> dict:
    """
    Get random quote

    Get randomly selected quote from database our programming quotes

    :return: selected quote
    :rtype: dict
    """

    return quotes[random.randint(0, len(quotes) - 1)]

So, a quote is selected by generating a random integer with random.randint between 0 and the last index.

Export the function in random_quote_generator/__init__.py:

"""
Random Quote Generator
======================

Get random quote from our database of programming wisdom
"""
from .get_quote import get_quote

__all__ = ["get_quote"]

The function is imported and listed inside the __all__ attribute, which is a list of public objects for the module. In other words, when someone uses from random_quote_generator import *, only names listed in __all__ will be imported.

The test should now pass:

$ poetry run python -m pytest tests

Add a .gitignore file:

__pycache__

Add the "random_quote_generator" and "tests" folders to git along with the .gitignore file:

$ git add random_quote_generator/ tests/ .gitignore

That's it. The package is ready for delivery.

Document the project

Our package works but our users will have to check it's source code to see how to use it. We've already included docstrings so we can easily create standalone project documentation with Sphinx.

If you're not familiar with docstrings or documentation as a standalone resource review the Documenting Python Code and Projects article.

Assuming you have Sphinx installed, run the following to scaffold out the files and folders for Sphinx in the project root:

$ sphinx-quickstart docs

You'll be promoted with some questions:

> Separate source and build directories (y/n) [n]: n
> Project name: Random Quote Generator
> Author name(s): Your Name
> Project release []: 0.1.0
> Project language [en]: en

Next, let's update the project config. Open docs/conf.py and replace this:

# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))

With this:

import os
import sys
sys.path.insert(0, os.path.abspath('..'))

Now, autodoc, which is used to pull in documentation from docstrings, will search for modules in the parent folder of "docs".

Add the following extensions to the extensions list:

extensions = [
    'sphinx.ext.autodoc',
]

Update docs/index.rst like so:

.. Random Quote Generator documentation master file, created by
   sphinx-quickstart on Mon Dec 21 22:27:23 2020.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to Random Quote Generator's documentation!
==================================================

.. automodule:: random_quote_generator
    :members:



Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

This file should be excluded from Flake8, which we'll be adding shortly. So create a .flake8 file in the project root:

[flake8]
exclude =
    docs/conf.py,

Add the "docs" folder and .flake8 to git:

$ git add docs/ .flake8

GitHub Actions

Next, let's wire up a CI pipeline with GitHub Actions.

Add the following files and folders to the project root:

.github
└── workflows
    └── branch.yaml

Inside branch.yaml, add:

name: Push
on: [push]

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        python-version: [3.9]
        poetry-version: [1.1.8]
        os: [ubuntu-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/[email protected]
      - uses: actions/[email protected]
        with:
          python-version: ${{ matrix.python-version }}
      - name: Run image
        uses: abatilo/[email protected]
        with:
          poetry-version: ${{ matrix.poetry-version }}
      - name: Install dependencies
        run: poetry install
      - name: Run tests
        run: poetry run pytest --cov=./ --cov-report=xml
      - name: Upload coverage to Codecov
        uses: codecov/[email protected]
  code-quality:
    strategy:
      fail-fast: false
      matrix:
        python-version: [3.9]
        poetry-version: [1.1.8]
        os: [ubuntu-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/[email protected]
      - uses: actions/[email protected]
        with:
          python-version: ${{ matrix.python-version }}
      - name: Run image
        uses: abatilo/[email protected]
        with:
          poetry-version: ${{ matrix.poetry-version }}
      - name: Install dependencies
        run: poetry install
      - name: Run black
        run: poetry run black . --check
      - name: Run isort
        run: poetry run isort . --check-only --profile black
      - name: Run flake8
        run: poetry run flake8 .
      - name: Run bandit
        run: poetry run bandit .
      - name: Run saftey
        run: poetry run safety check

This configuration:

  • runs on every push on every single branch - on: [push]
  • runs on the latest version of Ubuntu - ubuntu-latest
  • uses Python 3.9 - python-version: [3.9], python-version: ${{ matrix.python-version }}
  • uses Poetry version 1.1.8 - poetry-version: [1.1.8], poetry-version: ${{ matrix.poetry-version }}

There are two jobs defined: test and code-quality. As the names' suggest, the tests run in the test job while our code quality checks run in the code-quality job.

Now on every push to the GitHub repository, tests and code quality jobs will run.

Add ".github" to git:

$ git add .github/

Run all of the code quality checks:

$ poetry run black .
$ poetry run isort . --profile black
$ poetry run flake8 .
$ poetry run bandit .
$ poetry run safety check

Make sure to add any files that may have changed to git. Then, commit and push your changes to GitHub:

$ git commit -m 'Package ready'
$ git push -u origin main

You should see your workflow running on the "Actions" tab on your GitHub repository. Make sure it passes before moving on.

CodeCov

Next, we'll configure CodeCov to track code coverage. Navigate to http://codecov.io/, and log in with your GitHub account and find your repository.

Check out the Quick Start guide for help with getting up and running with CodeCov.

Run the GitHub Actions workflow again. Once done, you should be able to see the coverage report on CodeCov:

CodeCov

Now, every time your workflow runs, a coverage report will be generated and uploaded to CodeCov. You can analyze changes in coverage percentage for branches, commits, and pull requests, focusing on increases and decreases in coverage over time.

Read The Docs

We'll use Read the Docs to host our documentation. Navigate to https://readthedocs.org, and log in using your GitHub account.

Next, click on "Import a Project". After that, refresh your projects and add the random quote generator project. Open the project and navigate to the "Admin" section. Then, under "Advanced Settings", set the default branch to main. Don't forget to save your changes.

Read the Docs

It will take a few minutes for Read The Docs to build the docs. Once done, you should be able to view your project documentation at https://your-project-slug-on-readthedocs.readthedocs.io.

Read the Docs

By default the documentation will be rebuilt on every push to the main branch. With that, the only thing left is to publish your package to PyPI.

PyPI

Finally, in order to make the project "pip-installable", we'll publish it to PyPI.

Start by adding the following section to pyproject.toml so that the "random_quote_generator" module is included in the distribution to PyPI:

packages = [
    { include = "random_quote_generator" },
]

Example file:

[tool.poetry]
name = "random-quote-generator-9308"
packages = [
    { include = "random_quote_generator" },
]
version = "0.1.0"
description = ""
authors = ["Michael Herman <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]
pytest = "^6.2.1"
pytest-cov = "^2.10.1"
black = "^20.8b1"
isort = "^5.7.0"
flake8 = "^3.8.4"
bandit = "^1.7.0"
safety = "^1.10.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Add a new file called release.yaml to ".github/workflows":

name: Release
on:
  release:
    types:
      - created

jobs:
  publish:
    strategy:
      fail-fast: false
      matrix:
        python-version: [3.9]
        poetry-version: [1.1.8]
        os: [ubuntu-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/[email protected]
      - uses: actions/[email protected]
        with:
          python-version: ${{ matrix.python-version }}
      - name: Run image
        uses: abatilo/[email protected]
        with:
          poetry-version: ${{ matrix.poetry-version }}
      - name: Publish
        env:
          PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
        run: |
          poetry config pypi-token.pypi $PYPI_TOKEN
          poetry publish --build

So, when a new release is created, the package will be published to PyPI.

Next, we'll need to create a PyPI token. Create an account on PyPI if you don't already have one. Then, once signed in, click "Account settings" and add a new API token. Copy the token. You'll now need to add it to your GitHub repository's secrets. Within the repo, click the "Settings" tab, and then click "Secrets". Use PYPI_TOKEN for the secret name and the token value as the secret value.

Now you're ready to create your first release.

Add the release.yaml file to git as well as the updated pyproject.toml file, commit, and push:

$ git add .github/workflows/release.yaml pyproject.toml
$ git commit -m 'Ready for first release'
$ git push -u origin main

To create a new release, navigate to https://github.com/<username>/<project-name>/releases/. Click "Create a new release". Enter 0.1.0 for the tag and Initial Release for the title.

GitHub new release

This will trigger a new workflow on GitHub Actions. Once complete, you should see your package on PyPI.

You should now be able to install and use your package from PyPI:

>>> from random_quote_generator import get_quote
>>>
>>> print(get_quote())
{'quote': 'Any fool can write code that a computer can understand. Good programmers write code that humans can understand.', 'author': 'Martin Fowler'}

Conclusion

We set up a simple CI/CD pipeline for a Python package published to PyPI. Code on each branch is checked with tests and code quality. You can check code coverage inside CodeCov. On releases, new versions are deployed to PyPI and the docs are updated. This can simplify your life a lot.

Automated pipelines like this help ensure that your workflow stays the same from day to day. Nonetheless, you should still care about your team's working culture. Tests run on push but they must be present. Automated test runs won't help much if you don't have any tests to run. Automation also won't make much difference to the size of your changes. Try to keep them small, merging to main often. Small changes along with Test-Driven Development (TDD) and a CI/CD pipeline can make a huge difference to the quality of the software you're shipping. Just don't forget that it always starts and ends with team culture.

The Complete Python Guide:

  1. Modern Python Environments - dependency and workspace management
  2. Testing in Python
  3. Modern Test-Driven Development in Python
  4. Python Code Quality
  5. Python Type Checking
  6. Documenting Python Code and Projects
  7. Python Project Workflow (this article!)

Jan Giacomelli

Jan Giacomelli

Jan is a software engineer who lives in Ljubljana, Slovenia, Europe. He is co-founder of typless where he is leading engineering efforts. He loves working with Python and Django. When he's not writing code or deploying to AWS, he's probably skiing, windsurfing, or playing guitar.

Share this tutorial

Featured Course

Test-Driven Development with Python, Flask, and Docker

In this course, you'll learn how to set up a development environment with Docker in order to build and deploy a microservice powered by Python and Flask. You'll also apply the practices of Test-Driven Development with pytest as you develop a RESTful API.

Featured Course

Test-Driven Development with Python, Flask, and Docker

In this course, you'll learn how to set up a development environment with Docker in order to build and deploy a microservice powered by Python and Flask. You'll also apply the practices of Test-Driven Development with pytest as you develop a RESTful API.