Jan Giacomelli
Thus far, in this series, we've covered:
- Modern Python Environments - dependency and workspace management
- Testing in Python
- Modern Test-Driven Development in Python
- Python Code Quality
- Python Type Checking
- Documenting Python Code and Projects
In this article, you'll glue everything together as you develop a single project from start to finish. After developing the basic project, you'll:
- Wire up CI/CD with GitHub Actions
- Configure coverage reporting with CodeCov
- Publish the package to PyPi and the docs to Read the Docs
- Update PyPI and Read the Docs via GitHub Actions
Feel free to swap out GitHub Actions for a similar CI/CD tool like GitLab CI, Bitbucket Pipelines, or CircleCI.
Contents
Project Setup
Let's build a random quote generator to return a randomly selected quote from set of quotes.
Initialize project
First, let's create a new folder for our project:
$ mkdir random-quote-generator
$ cd random-quote-generator
Initialize the project with Poetry:
$ poetry init
Package name [random_quote_generator]:
Version [0.1.0]:
Description []:
Author [Your name <[email protected]>, n to skip]:
License []:
Compatible Python versions [^3.10]:
Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Do you confirm generation? (yes/no) [yes]
For more on Poetry, check out the Modern Python Environments - Dependency and workspace management article.
Your project name must be unique since you'll be uploading it to PyPI. So, to avoid name collisions add a unique string to the package name in pyproject.toml.
For example:
[tool.poetry]
name = "random-quote-generator-9308"
version = "0.1.0"
description = ""
authors = ["Michael Herman <[email protected]>"]
[tool.poetry.dependencies]
python = "^3.10"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
Create a new repository on GitHub:
Next, initialize a git repository inside your project:
$ git init
$ git add pyproject.toml
$ git commit -m "first commit"
$ git branch -M main
$ git remote add origin [email protected]:<your-github-username>/random-quote-generator.git
$ git fetch
$ git branch --set-upstream-to=origin/main main
$ git pull origin main --rebase
$ git push -u origin main
With the basic setup complete, let's continue by adding the following development dependencies:
Review the Python Code Quality article for details on these dependencies.
Install:
$ poetry add --dev pytest pytest-cov black isort flake8 bandit safety
Add the new poetry.lock file as well as the updated pyproject.toml file to git:
$ git add poetry.lock pyproject.toml
Build the Project
After that, create a new folder called "random_quote_generator". Inside that folder, add an __init__.py file, so it's treated as a module, along with a quotes.py file.
random-quote-generator
├── poetry.lock
├── pyproject.toml
└── random_quote_generator
├── __init__.py
└── quotes.py
Inside quotes.py, add:
quotes = [
{
"quote": "A long descriptive name is better than a short "
"enigmatic name. A long descriptive name is better "
"than a long descriptive comment.",
"author": "Robert C. Martin",
},
{
"quote": "You should name a variable using the same "
"care with which you name a first-born child.",
"author": "Robert C. Martin",
},
{
"quote": "Any fool can write code that a computer "
"can understand. Good programmers write code"
" that humans can understand.",
"author": "Martin Fowler",
},
]
Nothing special there. Just a list of dictionaries, one for each quote. Next, create a new folder in the project root called "tests" and add the following files:
tests
├── __init__.py
└── test_get_quote.py
test_get_quote.py:
from random_quote_generator import get_quote
from random_quote_generator.quotes import quotes
def test_get_quote():
"""
GIVEN
WHEN get_quote is called
THEN random quote from quotes is returned
"""
quote = get_quote()
assert quote in quotes
Run the test:
$ poetry run python -m pytest tests
It should fail:
E ImportError: cannot import name 'get_quote' from 'random_quote_generator'
Next, add a new file to "random_quote_generator" called get_quote.py:
import random
from random_quote_generator.quotes import quotes
def get_quote() -> dict:
"""
Get random quote
Get randomly selected quote from database our programming quotes
:return: selected quote
:rtype: dict
"""
return quotes[random.randint(0, len(quotes) - 1)]
So, a quote is selected by generating a random integer with random.randint between 0 and the last index.
Export the function in random_quote_generator/__init__.py:
"""
Random Quote Generator
======================
Get random quote from our database of programming wisdom
"""
from .get_quote import get_quote
__all__ = ["get_quote"]
The function is imported and listed inside the __all__ attribute, which is a list of public objects for the module. In other words, when someone uses from random_quote_generator import *, only names listed in __all__ will be imported.
The test should now pass:
$ poetry run python -m pytest tests
Create a .gitignore file in the root of your project:
__pycache__
Add the "random_quote_generator" and "tests" folders to git along with the .gitignore file:
$ git add random_quote_generator/ tests/ .gitignore
That's it. The package is ready for delivery.
Document the project
Our package works but our users will have to check it's source code to see how to use it. We've already included docstrings so we can easily create standalone project documentation with Sphinx.
If you're not familiar with docstrings or documentation as a standalone resource review the Documenting Python Code and Projects article.
Assuming you have Sphinx installed, run the following to scaffold out the files and folders for Sphinx in the project root:
$ sphinx-quickstart docs
You'll be promoted with some questions:
> Separate source and build directories (y/n) [n]: n
> Project name: Random Quote Generator
> Author name(s): Your Name
> Project release []: 0.1.0
> Project language [en]: en
Next, let's update the project config. Open docs/conf.py and replace this:
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
With this:
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
Now, autodoc, which is used to pull in documentation from docstrings, will search for modules in the parent folder of "docs".
Add the following extensions to the extensions list:
extensions = [
'sphinx.ext.autodoc',
]
Update docs/index.rst like so:
.. Random Quote Generator documentation master file, created by
sphinx-quickstart on Mon Dec 21 22:27:23 2020.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Random Quote Generator's documentation!
==================================================
.. automodule:: random_quote_generator
:members:
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
This file should be excluded from Flake8, which we'll be adding shortly. So create a .flake8 file in the project root:
[flake8]
exclude =
docs/conf.py,
Add the "docs" folder and .flake8 to git:
$ git add docs/ .flake8
GitHub Actions
Next, let's wire up a CI pipeline with GitHub Actions.
Add the following files and folders to the project root:
.github
└── workflows
└── branch.yaml
Inside branch.yaml, add:
name: Push
on: [push]
jobs:
test:
strategy:
fail-fast: false
matrix:
python-version: ['3.10']
poetry-version: ['1.1.13']
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Run image
uses: abatilo/[email protected]
with:
poetry-version: ${{ matrix.poetry-version }}
- name: Install dependencies
run: poetry install
- name: Run tests
run: poetry run pytest --cov=./ --cov-report=xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
code-quality:
strategy:
fail-fast: false
matrix:
python-version: ['3.10']
poetry-version: ['1.1.13']
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Run image
uses: abatilo/[email protected]
with:
poetry-version: ${{ matrix.poetry-version }}
- name: Install dependencies
run: poetry install
- name: Run black
run: poetry run black . --check
- name: Run isort
run: poetry run isort . --check-only --profile black
- name: Run flake8
run: poetry run flake8 .
- name: Run bandit
run: poetry run bandit .
- name: Run saftey
run: poetry run safety check
This configuration:
- runs on every push on every single branch -
on: [push] - runs on the latest version of Ubuntu -
ubuntu-latest - uses Python 3.10 -
python-version: [3.10],python-version: ${{ matrix.python-version }} - uses Poetry version 1.1.13 -
poetry-version: [1.1.13],poetry-version: ${{ matrix.poetry-version }}
There are two jobs defined: test and code-quality. As the names' suggest, the tests run in the test job while our code quality checks run in the code-quality job.
Now on every push to the GitHub repository, tests and code quality jobs will run.
Add ".github" to git:
$ git add .github/
Run all of the code quality checks:
$ poetry run black .
$ poetry run isort . --profile black
$ poetry run flake8 .
$ poetry run bandit .
$ poetry run safety check
Make sure to add any files that may have changed to git. Then, commit and push your changes to GitHub:
$ git add docs/ random_quote_generator/ tests/
$ git commit -m 'Package ready'
$ git push -u origin main
You should see your workflow running on the "Actions" tab on your GitHub repository. Make sure it passes before moving on.
CodeCov
Next, we'll configure CodeCov to track code coverage. Navigate to http://codecov.io/, and log in with your GitHub account and find your repository.
Check out the Quick Start guide for help with getting up and running with CodeCov.
Run the GitHub Actions workflow again. Once done, you should be able to see the coverage report on CodeCov:
Now, every time your workflow runs, a coverage report will be generated and uploaded to CodeCov. You can analyze changes in coverage percentage for branches, commits, and pull requests, focusing on increases and decreases in coverage over time.
Read The Docs
We'll use Read the Docs to host our documentation. Navigate to https://readthedocs.org, and log in using your GitHub account.
If you just signed up, make sure to verify your email address before continuing.
Next, click on "Import a Project". After that, refresh your projects and add the random quote generator project. Open the project and navigate to the "Admin" section. Then, under "Advanced Settings", set the default branch to main. Don't forget to save your changes.
It will take a few minutes for Read The Docs to build the docs. Once done, you should be able to view your project documentation at https://your-project-slug-on-readthedocs.readthedocs.io.
By default the documentation will be rebuilt on every push to the main branch. With that, the only thing left is to publish your package to PyPI.
PyPI
Finally, in order to make the project "pip-installable", we'll publish it to PyPI.
Start by adding the following section to pyproject.toml so that the "random_quote_generator" module is included in the distribution to PyPI:
packages = [
{ include = "random_quote_generator" },
]
Example file:
[tool.poetry]
name = "random-quote-generator-93618"
packages = [
{ include = "random_quote_generator" },
]
version = "0.1.0"
description = ""
authors = ["Amir Tadrisi <[email protected]>"]
[tool.poetry.dependencies]
python = "^3.10"
[tool.poetry.dev-dependencies]
pytest = "^7.1.2"
pytest-cov = "^3.0.0"
black = "^22.3.0"
isort = "^5.10.1"
flake8 = "^4.0.1"
bandit = "^1.7.4"
safety = "^1.10.3"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
Add a new file called release.yaml to ".github/workflows":
name: Release
on:
release:
types:
- created
jobs:
publish:
strategy:
fail-fast: false
matrix:
python-version: ['3.10']
poetry-version: ['1.1.13']
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Run image
uses: abatilo/[email protected]
with:
poetry-version: ${{ matrix.poetry-version }}
- name: Publish
env:
PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
run: |
poetry config pypi-token.pypi $PYPI_TOKEN
poetry publish --build
So, when a new release is created, the package will be published to PyPI.
Next, we'll need to create a PyPI token. Create an account on PyPI if you don't already have one. Then, once signed in, click "Account settings" and add a new API token. Copy the token. You'll now need to add it to your GitHub repository's secrets. Within the repo, click the "Settings" tab, and then click "Secrets" and choose "Actions". Use PYPI_TOKEN for the secret name and the token value as the secret value.
Now you're ready to create your first release.
Add the release.yaml file to git as well as the updated pyproject.toml file, commit, and push:
$ git add .github/workflows/release.yaml pyproject.toml
$ git commit -m 'Ready for first release'
$ git push -u origin main
To create a new release, navigate to https://github.com/<username>/<project-name>/releases/. Click "Create a new release". Enter 0.1.0 for the tag and Initial Release for the title.
This will trigger a new workflow on GitHub Actions. Once complete, you should see your package on PyPI.
You should now be able to install and use your package from PyPI:
>>> from random_quote_generator import get_quote
>>>
>>> print(get_quote())
{'quote': 'Any fool can write code that a computer can understand. Good programmers write code that humans can understand.', 'author': 'Martin Fowler'}
Conclusion
We set up a simple CI/CD pipeline for a Python package published to PyPI. Code on each branch is checked with tests and code quality. You can check code coverage inside CodeCov. On releases, new versions are deployed to PyPI and the docs are updated. This can simplify your life a lot.
Automated pipelines like this help ensure that your workflow stays the same from day to day. Nonetheless, you should still care about your team's working culture. Tests run on push but they must be present. Automated test runs won't help much if you don't have any tests to run. Automation also won't make much difference to the size of your changes. Try to keep them small, merging to main often. Small changes along with Test-Driven Development (TDD) and a CI/CD pipeline can make a huge difference to the quality of the software you're shipping. Just don't forget that it always starts and ends with team culture.
The Complete Python Guide: