Using Hypothesis and Schemathesis to Test FastAPI

Last updated October 22nd, 2020

Testing is a necessary part of developing software. Software projects with high test coverage are never perfect, but it's a good initial indicator of the quality of the software. To encourage the writing of tests, tests should be fun and easy to write. They should also be treated with the same care as any other code in your codebase. Thus, you need to take the cost of maintaining the test suite into account when adding new tests. It's not easy balancing readability and maintainability while also ensuring tests cover a wide range of scenarios

In this article, we'll look at how property-based testing can help with this. We'll start by looking at what property-based testing is and why you may want to use it. Then, well show how Hypothesis and Schemathesis can be used to apply property-based testing to FastAPI.

Contents

Property-based Testing

What is property-based testing?

Property-based tests are based on the properties of a given function or a program. These tests help ensure that the function or program under test abides by its properties.

Benefits

Why use property-based testing?

  1. Scope: Rather than having to write different test cases for every argument you want to test, property-based testing allows you to test a range of arguments for each parameter from a single test. This helps increase the robustness of your test suite while decreasing test redundancy. In short, your test code will be cleaner, more DRY, and overall more efficient while at the same time more effective since you'll be able to test all those edge cases much easier.
  2. Reproducibility: Testing agents save the test cases along with their results, which can be used for reproducing and replaying a test in case of failure.

Let's look at a quick example to help illustrate the point:

def factorial(num: int) -> int:
    if num < 0:
        raise ValueError("Number must be >= 0")

    total = 1
    for _ in range(1, num + 1):
        total *= _
    return total


# test

import pytest


def test_factorial_less_than_0():
    with pytest.raises(ValueError):
        assert factorial(-1) == 1


def test_factorial():
    assert factorial(0) == 1
    assert factorial(1) == 1
    assert factorial(3) == 6
    assert factorial(7) == 5040
    assert factorial(12) == 479001600
    assert factorial(44) == 2658271574788448768043625811014615890319638528000000000

What's wrong with this?

  1. test cases are boring to write
  2. random, unbiased test examples are hard to come up with
  3. the test suite will quickly balloon in size so it will be hard to read and maintain going forward
  4. again, it's boring!
  5. it's hard to flesh out edge cases

Humans should not waste their time on this. It's the perfect task for a computer to do.

Property-based Testing with Hypothesis

Hypothesis is a tool for conducting property-based testing in Python. Hypothesis makes it easy to write tests and find all edge cases.

It works by generating arbitrary data matching your specification and checking that your guarantee still holds in that case. If it finds an example where it doesn’t, it takes that example and cuts it down to size, simplifying it until it finds a much smaller example that still causes the problem. It then saves that example for later so that once it has found a problem with your code, it will not forget it in the future.

The most important part here is that all failed tests will be tried even after the errors have been fixed!

Quick Start

Hypothesis integrates into your normal pytest or unittest workflow.

Start by installing the library:

$ pip install hypothesis

Next, you'll need to define a Strategy, which is a recipe for generating random data.

Examples:

strategy What it generates
binary binary numbers
text strings
integers integers
fractions float

Strategies are meant to be composed together to generate complex input data for testing. Thus, rather than having to write and maintain your own data generators, Hypothesis manages all that for you.

Let's refactor test_factorial from above into a property-based test:

from hypothesis import given
from hypothesis.strategies import integers


@given(integers(min_value=1, max_value=30))
def test_factorial(num: int):
    result = factorial(num) / factorial(num - 1)
    assert num == result

This test now asserts that the factorial of a number divided by the factorial of that number minus one is the original number.

Here, we passed the integers Strategy to the @given decorator, which is the the entry point to Hypothesis. This decorator essentially turns the test function into a parameterized function so that when it's called the generated data from the Strategy will be passed into the test.

If a failure had been found, Hypothesis uses Shrinking to find the smallest fail case.

Example:

from hypothesis import Verbosity, given, settings
from hypothesis import strategies as st


@settings(verbosity=Verbosity.verbose)
@given(st.integers())
def test_shrinking(num: int):
    assert num >= -2

Test output:

...

Trying example: test_shrinking(
    num=-4475302896957925906,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=2872,
)
Trying example: test_shrinking(
    num=-93,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=14443,
)
Trying example: test_shrinking(
    num=56,
)
Trying example: test_shrinking(
    num=-13873,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=23519,
)
Trying example: test_shrinking(
    num=-91,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=-93,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=0,
)
Trying example: test_shrinking(
    num=0,
)
Trying example: test_shrinking(
    num=-29,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=-13,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=-5,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=-1,
)
Trying example: test_shrinking(
    num=-2,
)
Trying example: test_shrinking(
    num=-4,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=-3,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=3,
)
Trying example: test_shrinking(
    num=-3,
)
Traceback (most recent call last):
  File "shrinking.py", line 8, in test_shrinking
    assert num >= -2
AssertionError

Trying example: test_shrinking(
    num=0,
)
Trying example: test_shrinking(
    num=-1,
)
Trying example: test_shrinking(
    num=-2,
)
Trying example: test_shrinking(
    num=3,
)
Falsifying example: test_shrinking(
    num=-3,
)

Here, we tested the expression num >= -2 against a pool of integers. Hypothesis started with num = -4475302896957925906, a rather large number, as the first fail case. It then shrinked the values of num until Hypothesis found the value num = -3 as the smallest fail case.

Using Hypothesis with FastAPI

Hypothesis has proven to be a simple yet powerful testing tool. Let's see how we can use it with FastAPI.

# server.py

import uvicorn
from fastapi import FastAPI

app = FastAPI()


@app.get("/api/{s}")
def homepage(s: int):
    return {"message": s * s}


if __name__ == "__main__":
    uvicorn.run(app)

So, the /api/{s} route takes a URL parameter called s that should be an integer.

# test_server.py

from hypothesis import given, strategies as st
from fastapi.testclient import TestClient

from server import app


client = TestClient(app)


@given(st.integers())
def test_home(s):
    res = client.get(f"/api/{s}")

    assert res.status_code == 200
    assert res.json() == {"message": s * s}

Like before we used the integers Strategy to generate random integers, positive and negative, for testing.

Schemathesis

Schemathesis is a modern API testing tool based on the OpenAPI and GraphQL specifications. It uses Hypothesis under the hood to apply property-based testing to API schemas. In other words, given a schema, Schemathesis can automatically generate test cases for you. Since FastAPI is based on OpenAPI standards, Schemathesis works well with it.

If you run the above server.py file and navigate to http://localhost:8000/openapi.json, you should see the OpenAPI specification generated by FastAPI. It defines all the endpoints along with their input types. Using this spec, Schemathesis can be used to generate test data.

Quick Start with the CLI

Install:

$ pip install schemathesis

Once installed, the simplest way to run tests is via the schemathesis command. With Uvicorn running in one terminal window, open a new window and run:

$ schemathesis run http://localhost:8000/openapi.json

You should see:

============================ Schemathesis test session starts ============================
platform Darwin -- Python 3.8.5, schemathesis-2.6.0, hypothesis-5.37.3,
    hypothesis_jsonschema-0.18.0, jsonschema-3.2.0
rootdir: /hypothesis-examples
hypothesis profile 'default' ->
    database=DirectoryBasedExampleDatabase('/hypothesis-examples/.hypothesis/examples')
Schema location: http://localhost:8000/openapi.json
Base URL: http://localhost:8000/
Specification version: Open API 3.0.2
Workers: 1
collected endpoints: 1

GET /api/{s} .                                                                      [100%]

======================================== SUMMARY =========================================

Performed checks:
    not_a_server_error                    100 / 100 passed          PASSED

=================================== 1 passed in 0.43s ====================================

Notice how this only checked for not_a_server_error. Schemathesis has five built-in checks:

  1. not_a_server_error: response has 5xx HTTP status
  2. status_code_conformance: response status is not defined in the API schema
  3. content_type_conformance: response content type is not defined in the API schema
  4. response_schema_conformance: response content does not conform to the schema defined for this specific response
  5. response_headers_conformance: response headers does not contain all defined headers.

You can perform all built-in checks with the --checks all option:

$ schemathesis run --checks all http://localhost:8000/openapi.json

============================ Schemathesis test session starts ============================
platform Darwin -- Python 3.8.5, schemathesis-2.6.0, hypothesis-5.37.3,
    hypothesis_jsonschema-0.18.0, jsonschema-3.2.0
rootdir: /hypothesis-examples
hypothesis profile 'default' ->
    database=DirectoryBasedExampleDatabase('/hypothesis-examples/.hypothesis/examples')
Schema location: http://localhost:8000/openapi.json
Base URL: http://localhost:8000/
Specification version: Open API 3.0.2
Workers: 1
collected endpoints: 1

GET /api/{s} .                                                                      [100%]

======================================== SUMMARY =========================================

Performed checks:
    not_a_server_error                              100 / 100 passed          PASSED
    status_code_conformance                         100 / 100 passed          PASSED
    content_type_conformance                        100 / 100 passed          PASSED
    response_headers_conformance                    100 / 100 passed          PASSED
    response_schema_conformance                     100 / 100 passed          PASSED

=================================== 1 passed in 0.41s ====================================

Additional Options

You can test a specific endpoint or HTTP method rather than the whole application:

$ schemathesis run --endpoint /api/. http://localhost:8000/openapi.json

$ schemathesis run --method GET http://localhost:8000/openapi.json

A max response time can be used to help flesh out edge cases that may slow down the endpoints. The time is in milliseconds.

$ schemathesis run --max-response-time=50 HTTP://localhost:8000/openapi.json

Do some of your endpoints require authorization?

$ schemathesis run -H "Authorization: Bearer TOKEN" http://localhost:8000/openapi.json

$ schemathesis run -H "Authorization: ..." -H "X-API-Key: ..." HTTP://localhost:8000/openapi.json

You can use multiple workers to speed up the tests:

$ schemathesis run --workers 8 http://localhost:8000/openapi.json

Normally, Schemathesis generates random data for each endpoint. Stateful tests make sure that the data comes from previous tests/responses:

$ schemathesis run --stateful=links http://localhost:8000/openapi.json

Finally, replaying tests is simple since each test case is associated with a seed value. When a test case fails, it'll provide the seed so that the you can reproduce the failed case:

$ schemathesis run http://localhost:8000/openapi.json

============================ Schemathesis test session starts ============================
platform Darwin -- Python 3.8.5, schemathesis-2.6.0, hypothesis-5.37.3,
    hypothesis_jsonschema-0.18.0, jsonschema-3.2.0
rootdir: /hypothesis-examples
hypothesis profile 'default' ->
    database=DirectoryBasedExampleDatabase('/hypothesis-examples/.hypothesis/examples')
Schema location: http://localhost:8000/openapi.json
Base URL: http://localhost:8000/
Specification version: Open API 3.0.2
Workers: 1
collected endpoints: 1

GET /api/{s} F                                                                      [100%]

======================================== FAILURES ========================================
_____________________________________ GET: /api/{s} ______________________________________
1. Received a response with 5xx status code: 500

Path parameters : {'s': 0}

Run this Python code to reproduce this failure:
  requests.get('http://localhost:8000/api/0', headers={'User-Agent': 'schemathesis/2.6.0'})

Or add this option to your command line parameters:
    --hypothesis-seed=135947773389980684299156880789978283847
======================================== SUMMARY =========================================

Performed checks:
    not_a_server_error                    0 / 3 passed          FAILED

=================================== 1 passed in 0.10s ====================================

Then, to reproduce, run:

$ schemathesis run http://localhost:8000/openapi.json --hypothesis-seed=135947773389980684299156880789978283847

Python Tests

You can use Schemathesis inside your tests as well:

import schemathesis

schema = schemathesis.from_uri("http://localhost:8000/openapi.json")


@schema.parametrize()
def test_api(case):
    case.call_and_validate()

Schemathesis also supports making calls directly to ASGI (i.e., Uvicorn and Daphne) and WSGI (i.e., Gunicorn and uWSGI) applications instead of over the network:

import schemathesis
from schemathesis.loaders import from_asgi

from server import app

schema = from_asgi("/openapi.json", app)


@schema.parametrize()
def test_api(case):
    response = case.call_asgi()
    case.validate_response(response)

Conclusion

Hopefully you can see just how powerful property-based testing can be. It can make your code more robust without reducing test readability from boilerplate test code. Features like shrinking to find the simplest failure cases and replaying improve productivity. Property-based testing will decrease time spent on writing manual tests while increasing test coverage.

A property-based testing tool like Hypothesis should be part of nearly every Python workflow. Schemathesis is used for automated API testing based on OpenAPI standards, which is very useful when coupled with an API-focused framework like FastAPI.

Amal Shaji

Amal Shaji

Amal is a full-stack developer interested in deep learning for computer vision and autonomous vehicles. He enjoys working with Python, PyTorch, Go, FastAPI, and Docker. He writes to learn and is a professional introvert.

Share this tutorial

Featured Course

Test-Driven Development with FastAPI and Docker

In this course, you'll learn how to build, test, and deploy a text summarization service with Python, FastAPI, and Docker. The service itself will be exposed via a RESTful API and deployed to Heroku with Docker.

Featured Course

Test-Driven Development with FastAPI and Docker

In this course, you'll learn how to build, test, and deploy a text summarization service with Python, FastAPI, and Docker. The service itself will be exposed via a RESTful API and deployed to Heroku with Docker.