Python Type Checking

Last updated November 24th, 2020

What is type checking? Why do we need it? What's the difference between static and runtime type checking?

Python is a strongly typed, dynamic programming language. With it being dynamically typed, types are dynamically inferred, so you can set variable values directly without defining the variable type like in statically typed programming languages such as Java.

name = "Michael"
String name = "Michael";

static vs dynamic programming languages

Strong and dynamic means that types are inferred at runtime but you can't mix types. For example, a = 1 + '0' will raise an error in Python. On the other hand, JavaScript is weak and dynamic, so types are inferred at runtime and you can mix types. For example, a = 1 + '0' will set a to 10.

While dynamic typing brings flexibility, it isn't always desirable so, there have been a number of efforts as of late to bring static type inference to dynamic languages.

In this article, we'll look at what type hints are and how they can benefit you. We'll also dive into how you can use Python's type system for static type checking with mypy and runtime type checking with pydantic, marshmallow, and typeguard.

The Complete Python Guide:

  1. Modern Python Environments - dependency and workspace management
  2. Testing in Python
  3. Modern Test-Driven Development in Python
  4. Python Code Quality
  5. Python Type Checking (this article!)
  6. Documenting Python Code and Projects
  7. Python Project Workflow

Contents

Tools

There are a number of tools out there that use type hints for static and runtime type checking.

Static typing

  1. mypy
  2. pyre
  3. Pyright
  4. pytype
  5. pyanalyze

Runtime type checking / data validation

  1. marshmallow
  2. pydantic
  3. typeguard
  4. typical
  5. pytypes

Project specific

  1. pydantic-django
  2. django-stubs
  3. typeddjango
  4. flask-pydantic
  5. flask-marshmallow
  6. fastapi (pydantic is built in -- yay!)

Check out Awesome Python Typing for a full list of tools.

Type Hints

Type hints were added to Python in version version 3.5.

They allow developers to annotate expected types for variables, function parameters, and function returns inside Python code. While such types are not enforced by the Python interpreter -- again, Python is a dynamically typed language -- they do offer a number of benefits. First and foremost, with type hints, you can better express the intent of what it is that your code is doing and how to use it. Better understanding results in fewer bugs.

For example, say you have the following function to calculate the average daily temperature:

def daily_average(temperatures):
    return sum(temperatures) / len(temperatures)

As long as you provide a list of temperatures like so, the function works as intended and it will return the expected result:

average_temperature = daily_average([22.8, 19.6, 25.9])
print(average_temperature)  # => 22.76666666666667

What happens if you call the function with a dictionary where keys are timestamps of the measurements and values are temperatures?

average_temperature = daily_average({1599125906: 22.8, 1599125706: 19.6, 1599126006: 25.9})
print(average_temperature)  # => 1599125872.6666667

Essentially, this function now returns the sum of keys / number of keys, which is clearly wrong. Since the function call didn't raise an error, this can go undetected especially if the end user provides the temperatures.

To avoid such confusion, you can add type hints by annotating the argument and return value:

from typing import List


def daily_average(temperatures: List[float]) -> float:
    return sum(temperatures) / len(temperatures)

Now the function definition tells us:

  1. temperatures should be list of floats: temperatures: List[float]
  2. the function should return a float: -> float
print(daily_average.__annotations__)
# {'temperatures': typing.List[float], 'return': <class 'float'>}

Type hints enable static type checking tools. Code editors and IDEs use them as well, warning you when usage of a particular function or method is not as expected according to the type hints and providing powerful autocompletion.

So, type hints are really just "hints". They are not as strict as type definitions in statically typed languages, in other words. That said, even though they are quite flexible, they still help to improve code quality by expressing intentions more clearly. Besides that you can use a lot of tools to benefit from them even more.

Type Annotations vs Type Hints

Type annotations are just syntax to annotate function inputs, function outputs, and variables:

def sum_xy(x: 'an integer', y: 'another integer') -> int:
    return x + y


print(sum_xy.__annotations__)
# {'x': 'an integer', 'y': 'another integer', 'return': <class 'int'}

Type hints are built on top of annotations to make them more useful. Hints and annotations are often used interchangeably but they are different.

Python's typing Module

You may be wondering why we can use the built-in float to define the function return type, but the List is imported from the typing module:

from typing import List


def daily_average(temperatures: List[float]) -> float:
    return sum(temperatures) / len(temperatures)

Before Python 3.9, the Python interpreter didn't support the use of built-ins with arguments for type hinting.

For example, it was possible to use list as a type hint like so:

def daily_average(temperatures: list) -> float:
    return sum(temperatures) / len(temperatures)

But it wasn't possible to define the expected type of list elements (List[float]) without the typing module. The same can be said for dictionaries and other sequences and complex types:

from typing import Tuple, Dict


def generate_map(points: Tuple[float, float]) -> Dict[str, int]:
    return map(points)

Besides that, the typing module allows you to define new types, type aliases, type Any and many other things.

For example, you may want to allow multiple types. For that you can use Union:

from typing import Union


def sum_ab(a: Union[int, float], b: Union[int, float]) -> Union[int, float]:
    return a + b

Since Python 3.9 you can use built-ins like so:

def sort_names(names: list[str]) -> list[str]:
    return sorted(names)

Static Type Checking with mypy

mypy is a tool for type checking at compile-time.

You can install it like any other Python package:

$ pip install mypy

To check your Python module you can run it like so:

$ python -m mypy my_module.py

So, let's look at the daily_average example again:

def daily_average(temperatures):
    return sum(temperatures) / len(temperatures)


average_temperature = daily_average(
    {1599125906: 22.8, 1599125706: 19.6, 1599126006: 25.9}
)

When type checking with mypy on such code, no errors will be reported since the function doesn't use type hints:

Success: no issues found in 1 source file

Add the type hints in:

from typing import List


def daily_average(temperatures: List[float]) -> float:
    return sum(temperatures) / len(temperatures)


average_temperature = daily_average(
    {1599125906: 22.8, 1599125706: 19.6, 1599126006: 25.9}
)

Run mypy again:

$ python -m mypy my_module.py

You should see:

my_module.py:9: error: Argument 1 to "daily_average" has incompatible
type "Dict[int, float]"; expected "List[float]"

Found 1 error in 1 file (checked 1 source file)

mypy recognized that the function was being called incorrectly. It reported the file name, line number, and description of the error. Using type hints in conjunction with mypy can help reduce the number of errors resulting from the misuse of functions, methods, and classes. This results in quicker feedback loops. You don't need to run all of your tests or even deploy the whole application. You're notified about such errors immediately.

It's also a good idea to add mypy to your CI pipeline as well to check typing before your code is merged or deployed. For more on this, review the Python Code Quality article.

Although it's a big improvement in terms of code quality, static type checking doesn't enforce types at runtime, as your program is running. That's why we also have runtime type checkers, which we'll look at next.

mypy comes with typeshed which contains external type annotations for the Python standard library and Python builtins as well as third-party packages.

mypy checks Python programs with basically no runtime overhead. Although it checks types, duck typing still happens. Therefore, it cannot be used to compile CPython extensions.

Runtime Type Checking

pydantic

Static type checkers don't help when dealing with data from external sources like the users of your application. That's where runtime type checkers come into play. One such tools is pydantic, which is used to validate data. It raises validation errors when the provided data does not match a type defined with a type hint.

pydantic uses type casting to convert input data to force it to conform to the expected type.

$ pip install pydantic

It's actually quite simple to use. For example, let's define a Song class with a few attributes:

from datetime import date
from typing import List

from pydantic import BaseModel


class Song(BaseModel):
    id: int
    name: str
    release: date
    genres: List[str]

Now, when we initialize a new Song with valid data, everything works as expected:

song = Song(
    id=101,
    name='Bohemian Rhapsody',
    release='1975-10-31',
    genres=[
        'Hard Rock',
        'Progressive Rock'
    ]
)
print(song)
# id=101 name='Bohemian Rhapsody' release=datetime.date(1975, 10, 31)
# genres=['Hard Rock', 'Progressive Rock']

However, when we try to initialize a new Song with invalid data ('1975-31-31'), a ValidationError is raised:

song = Song(
    id=101,
    name='Bohemian Rhapsody',
    release='1975-31-31',
    genres=[
        'Hard Rock',
        'Progressive Rock'
    ]
)
print(song)
# pydantic.error_wrappers.ValidationError: 1 validation error for Song
# release
#   invalid date format (type=value_error.date)

With pydantic, we can ensure that only data that matches the defined types are used in our application. This results not only in fewer bugs but you'll need to write fewer tests as well. By using tools such as pydantic we don't need to write tests for cases where the user sends completely wrong data. It's handled by pydantic -- a ValidationError is raised. For example, FastAPI validates HTTP request and response bodies with pydantic:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


class Item(BaseModel):
    name: str
    price: float


@app.post("/items/", response_model=Item)
async def create_item(item: Item):
    return item

The create_item handler expects a payload with a name (string) and price (float). The response object should look the same. Now, if there's an issue with the provided payload, an error is raised right away. Raising it late makes it harder to debug and to determine where the data of the wrong type came from. Plus, since it's handled by pydantic, you can keep your route handlers clean.

Along with leveraging type hints for data validation, you can also add custom validators to ensure correctness of data beyond its type. Adding custom validation for an attribute is fairly easy. For example, to prevent genre duplications in the Song class, you can add validation like so:

from datetime import date
from typing import List

from pydantic import BaseModel, validator


class Song(BaseModel):
    id: int
    name: str
    release: date
    genres: List[str]

    @validator('genres')
    def no_duplicates_in_genre(cls, v):
        if len(set(v)) != len(v):
            raise ValueError(
                'No duplicates allowed in genre.'
            )
        return v


song = Song(
    id=101,
    name='Bohemian Rhapsody',
    release='1975-10-31',
    genres=[
        'Hard Rock',
        'Progressive Rock',
        'Progressive Rock',
    ]
)
print(song)
# pydantic.error_wrappers.ValidationError: 1 validation error for Song
# genre
#   No duplicates allowed in genre. (type=value_error)

So, the validation method, no_duplicates_in_genre, must be decorated with validator, which takes the attribute name as an argument. The validation method must be a class method since validation happens before the instance is created. For data that fails validation it should raise a standard Python ValueError.

You can also use validator methods to alter the value before validation occurs. To do so, add pre=True and always=True to the validator decorator:

@validator('genres', pre=True, always=True)

For example, you can convert genres to lower case like so:

from datetime import date
from typing import List

from pydantic import BaseModel, validator


class Song(BaseModel):
    id: int
    name: str
    release: date
    genres: List[str]

    @validator('genres', pre=True, always=True)
    def to_lower_case(cls, v):
        return [genre.lower() for genre in v]

    @validator('genres')
    def no_duplicates_in_genre(cls, v):
        if len(set(v)) != len(v):
            raise ValueError(
                'No duplicates allowed in genre.'
            )
        return v


song = Song(
    id=101,
    name='Bohemian Rhapsody',
    release='1975-10-31',
    genres=[
        'Hard Rock',
        'PrOgReSsIvE ROCK',
        'Progressive Rock',
    ]
)
print(song)
# pydantic.error_wrappers.ValidationError: 1 validation error for Song
# genre
#   No duplicates allowed in genre. (type=value_error)

to_lower_case converts every element in the genres list to lowercase. Because pre is set to True this method is called before pydantic validates the types. All generes are converted to lowercase and then validated with no_duplicates_in_genre.

pydantic also offers more strict types like PositiveInt and EmailStr to make your validations even better. Review Field Types from the docs for more on this.

Marshmallow

Another tool worth mentioning is marshmallow, which helps to validate complex data and load/dump data from/to native Python types. Installation is the same as for any other Python package:

$ pip install marshmallow

Like pydantic, you can add type validation to a class:

from marshmallow import Schema, fields, post_load


class Song:
    def __init__(
            self,
            id,
            name,
            release,
            genres
    ):
        self.id = id
        self.name = name
        self.release = release
        self.genres = genres

    def __repr__(self):
        return (
            f'<Song(id={self.id}, name={self.name}), '
            f'release={self.release.isoformat()}, genres={self.genres}>'
        )


class SongSchema(Schema):
    id = fields.Int()
    name = fields.Str()
    release = fields.Date()
    genres = fields.List(fields.String())

    @post_load
    def make_song(self, data, **kwargs):
        return Song(**data)


external_data = {
    'id': 101,
    'name': 'Bohemian Rhapsody',
    'release': '1975-10-31',
    'genres': ['Hard Rock', 'Progressive Rock']
}

song = SongSchema().load(external_data)
print(song)
# <Song(id=101, name=Bohemian Rhapsody), release=1975-10-31, genres=['Hard Rock', 'Progressive Rock']>

Unlike pydantic, marshmallow doesn't use type casting, so you need to define the schema and class separately. For example, release date in external_data must be an ISO string. It doesn't work with a datetime object.

To enable deserializing data into a Song object, you need to add a method decorated with @post_load decorator to the schema:

class SongSchema(Schema):
    id = fields.Int()
    name = fields.Str()
    release = fields.Date()
    genres = fields.List(fields.String(), validate=no_duplicates)

    @post_load
    def make_song(self, data, **kwargs):
        return Song(**data)

The schema validates the data and if all fields are valid, it creates an instance of the class by calling make_song with the validated data.

Like pydantic, you can add custom validations for each attribute from the schema. For example, you can prevent duplicates like so:

import datetime

from marshmallow import Schema, fields, post_load, ValidationError


class Song:
    def __init__(
            self,
            id,
            name,
            release,
            genres
    ):
        self.id = id
        self.name = name
        self.release = release
        self.genres = genres

    def __repr__(self):
        return (
            f'<Song(id={self.id}, name={self.name}), '
            f'release={self.release.isoformat()}, genres={self.genres}>'
        )


def no_duplicates(genres):
    if isinstance(genres, list):
        genres = [
            genre.lower()
            for genre in genres
            if isinstance(genre, str)
        ]

        if len(set(genres)) != len(genres):
            raise ValidationError(
                'No duplicates allowed in genres.'
            )


class SongSchema(Schema):
    id = fields.Int()
    name = fields.Str()
    release = fields.Date()
    genres = fields.List(fields.String(), validate=no_duplicates)

    @post_load
    def make_song(self, data, **kwargs):
        return Song(**data)


external_data = {
    'id': 101,
    'name': 'Bohemian Rhapsody',
    'release': '1975-10-31',
    'genres': ['Hard Rock', 'Progressive Rock', 'ProgressivE Rock']
}

song = SongSchema().load(external_data)
print(song)
# marshmallow.exceptions.ValidationError:
# {'genres': ['No duplicates allowed in genres.']}

As you can see, you can use either pydantic or marshmallow to ensure data has the correct type as your application runs. Pick the one that fits your style better.

Typeguard

While pydantic and marshmallow focus on data validation and serialization, typeguard focuses on checking types as functions are called. While mypy just does static type checking, typeguard enforces types while your program is running.

$ pip install typeguard

Let's take a look at the same example as before -- a Song class. This time we define it's __init__ method with type hinted arguments:

from datetime import date
from typing import List

from typeguard import typechecked


@typechecked
class Song:

    def __init__(
            self,
            id: int,
            name: str,
            release: date,
            genres: List[str]

    ) -> None:
        self.id = id
        self.name = name
        self.release = release
        self.genres = genres


song = Song(
    id=101,
    name='Bohemian Rhapsody',
    release=date(1975, 10, 31),
    genres={
        'Hard Rock',
        'Progressive Rock',
    }
)
print(song)
# TypeError: type of argument "genres" must be a list; got set instead

The typechecked decorator can be used for both classes and functions when you want to enforce type checking during runtime. Running this code will raise a TypeError since genres are a set instead of a list. You can similarly use a decorator for functions like so:

from typeguard import typechecked

@typechecked
def sum_ab(a: int, b: int) -> int:
    return a + b

It also comes with a pytest plugin. To check types for package my_package while running tests you can run this command:

$ python -m pytest --typeguard-packages=my_package

When running with pytest you don't need to use the @typechecked decorator. So, you can either decorate your functions and classes to enforce types during runtime or just during test runs. Either way, typeguard can be a powerful safety net for your application to ensure it runs as expected.

Flask with pydantic

So let's put it all together into a web application. As mentioned above, FastAPI uses pydantic by default. Although Flask doesn't have built-in support for pydantic we can use bindings to also add it to our APIs. So let's create a new Flask project to see it in action.

First, create a new folder:

$ mkdir flask_example
$ cd flask_example

Next, initialize your project with Poetry:

$ poetry init
Package name [flask_example]:
Version [0.1.0]:
Description []:
Author [Your name <[email protected]>, n to skip]:
License []:
Compatible Python versions [^3.7]:  >3.7

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Do you confirm generation? (yes/no) [yes]

After that, add Flask, Flask-Pydantic, and pytest:

$ poetry add flask Flask-Pydantic
$ poetry add --dev pytest

Create a file to hold our tests called test_app.py:

import json

import pytest

from app import app


@pytest.fixture
def client():
    app.config["TESTING"] = True

    with app.test_client() as client:
        yield client


def test_create_todo(client):
    response = client.post(
        "/todos/",
        data=json.dumps(
            {
                'title': 'Wash the dishes',
                'done': False,
                'deadline': '2020-12-12'
            }
        ),
        content_type='application/json'
    )

    assert response.status_code == 201


def test_create_todo_bad_request(client):
    response = client.post(
        "/todos/",
        data=json.dumps(
            {
                'title': 'Wash the dishes',
                'done': False,
                'deadline': 'WHENEVER'
            }
        ),
        content_type='application/json'
    )

    assert response.status_code == 400

Here, we have two tests for creating new todos. One checks that a status of 201 is returned when everything is fine. Another checks that a status of 400 is returned when the provided data is not as expected.

Next, add a file for the Flask app called app.py:

import datetime

from flask import Flask, request
from flask_pydantic import validate
from pydantic import BaseModel

app = Flask(__name__)


class CreateTodo(BaseModel):
    title: str
    done: bool
    deadline: datetime.date


class Todo(BaseModel):
    title: str
    done: bool
    deadline: datetime.date
    created_at: datetime.datetime


@app.route("/todos/", methods=['POST'])
@validate(body=CreateTodo)
def todos():
    todo = Todo(
        title=request.body_params.title,
        done=request.body_params.done,
        deadline=request.body_params.deadline,
        created_at=datetime.datetime.now()
    )

    return todo, 201


if __name__ == "__main__":
    app.run()

We've defined an endpoint for creating todos along with a request schema called CreateTodo and a response schema called Todo. Now when data is sent to the API that does not match the request schema a status of 400 with validation errors in the body is returned. You can run tests now to check that your API is actually behaving as described:

$ poetry run pytest

Running Type Checkers

Now that you know the tools, the next question is: When should they be used?

Much like code quality tools, you typically run type checkers:

  1. While coding (inside your IDE or code editor)
  2. At commit time (with pre-commit hooks)
  3. When code is checked in to source control (via a CI pipeline)
  4. During program run (runtime checkers)

Inside Your IDE or Code Editor

It's best to check for issues that could have a negative impact on quality early and often. Therefore, it's recommended to statically check your code during development. Many of the popular IDEs have mypy or mypy-like static type checkers built in. For those that don't, there's likely a plugin available. Such plugins warn you in real-time about type violations and potential programming errors.

Resources:

  1. Adding type hints in PyCharm
  2. Type hints in Visual Studio Code
  3. Sublime Text Package Finder
  4. Atom Packages

Pre-commit Hooks

Since you'll inevitably miss a warning here and there as you're coding, it's a good practice to check for static type issues at commit time with pre-commit git hooks. This way you can avoid committing code that won't pass type checks inside your CI pipeline.

The pre-commit framework is recommended for managing git hooks.

$ pip install pre-commit

Once installed, add a pre-commit config file called .pre-commit-config.yaml to your project. To run mypy, add the following config:

repos:
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: 'v0.790'
    hooks:
      - id: mypy

Finally, to set up the git hook scripts:

(venv)$ pre-commit install

Now, every time you run git commit mypy will run before the actual commit is made. And if there are any issues, the commit will be aborted.

CI Pipeline

It makes sense to run static type checks inside your CI pipeline to prevent type issues to be merged into the code base. This is probably the most important time to run mypy or some other static type checker.

You may experience issues when running static type checks with mypy, especially when using third-party libraries without type hints. That's probably the major reason why many people avoid running mypy checks inside the CI pipeline.

During Program Run

All previous times to run are before your program is actually running. That's the job for static type checkers. For dynamic type checkers you need a running program. As mentioned before, using them will require less tests, produce less bugs, and help you detect bugs early. You can use them for data validations (with pydantic and marshmallow) and to enforce types during program run (with typeguard).

Conclusion

Type checking may seem unnecessary when a code base is small, but the larger it gets, the more important it is. It's one more layer that protects us against easily preventable bugs. Type hints, although they're not enforced by the interpreter, help to better express the intent of a variable, function, or class. Most modern IDEs and code editors provide plugins to notify developers about type mismatches based on type hints. To enforce them, we can include mypy into our workflow to statically check whether usage of methods match their type hints. Although static analysis can improve your code, you must take in account that our software is communicating with the external world. Because of that, it's encouraged to add runtime type checkers like pydantic or marshmallow. They help with validating user input and raise errors at the earliest possible stage. The faster you find an error, the easier it is to correct it and move on.

The Complete Python Guide:

  1. Modern Python Environments - dependency and workspace management
  2. Testing in Python
  3. Modern Test-Driven Development in Python
  4. Python Code Quality
  5. Python Type Checking (this article!)
  6. Documenting Python Code and Projects
  7. Python Project Workflow

Jan Giacomelli

Jan Giacomelli

Jan is a software engineer who lives in Ljubljana, Slovenia, Europe. He is co-founder of typless where he is leading engineering efforts. He loves working with Python and Django. When he's not writing code or deploying to AWS, he's probably skiing, windsurfing, or playing guitar.

Share this tutorial

Featured Course

Building Your Own Python Web Framework

In this course, you'll learn how to develop your own Python web framework to see how all the magic works beneath the scenes in Flask, Django, and the other Python-based web frameworks.

Featured Course

Building Your Own Python Web Framework

In this course, you'll learn how to develop your own Python web framework to see how all the magic works beneath the scenes in Flask, Django, and the other Python-based web frameworks.