Amal Shaji
Testing is a necessary part of developing software. Software projects with high test coverage are never perfect, but it's a good initial indicator of the quality of the software. To encourage the writing of tests, tests should be fun and easy to write. They should also be treated with the same care as any other code in your codebase. Thus, you need to take the cost of maintaining the test suite into account when adding new tests. It's not easy balancing readability and maintainability while also ensuring tests cover a wide range of scenarios
In this article, we'll look at how property-based testing can help with this. We'll start by looking at what property-based testing is and why you may want to use it. Then, we'll show how Hypothesis and Schemathesis can be used to apply property-based testing to FastAPI.
Contents
Property-based Testing
What is property-based testing?
Property-based tests are based on the properties of a given function or a program. These tests help ensure that the function or program under test abides by its properties.
Benefits
Why use property-based testing?
- Scope: Rather than having to write different test cases for every argument you want to test, property-based testing allows you to test a range of arguments for each parameter from a single test. This helps increase the robustness of your test suite while decreasing test redundancy. In short, your test code will be cleaner, more DRY, and overall more efficient while at the same time more effective since you'll be able to test all those edge cases much easier.
- Reproducibility: Testing agents save the test cases along with their results, which can be used for reproducing and replaying a test in case of failure.
Let's look at a quick example to help illustrate the point:
def factorial(num: int) -> int:
if num < 0:
raise ValueError("Number must be >= 0")
total = 1
for _ in range(1, num + 1):
total *= _
return total
# test
import pytest
def test_factorial_less_than_0():
with pytest.raises(ValueError):
assert factorial(-1) == 1
def test_factorial():
assert factorial(0) == 1
assert factorial(1) == 1
assert factorial(3) == 6
assert factorial(7) == 5040
assert factorial(12) == 479001600
assert factorial(44) == 2658271574788448768043625811014615890319638528000000000
What's wrong with this?
- test cases are boring to write
- random, unbiased test examples are hard to come up with
- the test suite will quickly balloon in size so it will be hard to read and maintain going forward
- again, it's boring!
- it's hard to flesh out edge cases
Humans should not waste their time on this. It's the perfect task for a computer to do.
Property-based Testing with Hypothesis
Hypothesis is a tool for conducting property-based testing in Python. Hypothesis makes it easy to write tests and find all edge cases.
It works by generating arbitrary data matching your specification and checking that your guarantee still holds in that case. If it finds an example where it doesn't, it takes that example and cuts it down to size, simplifying it until it finds a much smaller example that still causes the problem. It then saves that example for later so that once it has found a problem with your code, it will not forget it in the future.
The most important part here is that all failed tests will be tried even after the errors have been fixed!
Quick Start
Hypothesis integrates into your normal pytest or unittest workflow.
Start by installing the library:
$ pip install hypothesis
Next, you'll need to define a Strategy, which is a recipe for generating random data.
Examples:
| strategy | What it generates |
|---|---|
| binary | bytestrings |
| text | strings |
| integers | integers |
| floats | floats |
| fractions | Fraction instances |
Strategies are meant to be composed together to generate complex input data for testing. Thus, rather than having to write and maintain your own data generators, Hypothesis manages all that for you.
Let's refactor test_factorial from above into a property-based test:
from hypothesis import given
from hypothesis.strategies import integers
@given(integers(min_value=1, max_value=30))
def test_factorial(num: int):
result = factorial(num) / factorial(num - 1)
assert num == result
This test now asserts that the factorial of a number divided by the factorial of that number minus one is the original number.
Here, we passed the integers Strategy to the @given decorator, which is the the entry point to Hypothesis. This decorator essentially turns the test function into a parameterized function so that when it's called the generated data from the Strategy will be passed into the test.
If a failure had been found, Hypothesis uses Shrinking to find the smallest fail case.
Example:
from hypothesis import Verbosity, given, settings
from hypothesis import strategies as st
@settings(verbosity=Verbosity.verbose)
@given(st.integers())
def test_shrinking(num: int):
assert num >= -2
Test output:
...
Trying example: test_shrinking(
num=-4475302896957925906,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=2872,
)
Trying example: test_shrinking(
num=-93,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=14443,
)
Trying example: test_shrinking(
num=56,
)
Trying example: test_shrinking(
num=-13873,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=23519,
)
Trying example: test_shrinking(
num=-91,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=-93,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=0,
)
Trying example: test_shrinking(
num=0,
)
Trying example: test_shrinking(
num=-29,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=-13,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=-5,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=-1,
)
Trying example: test_shrinking(
num=-2,
)
Trying example: test_shrinking(
num=-4,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=-3,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=3,
)
Trying example: test_shrinking(
num=-3,
)
Traceback (most recent call last):
File "shrinking.py", line 8, in test_shrinking
assert num >= -2
AssertionError
Trying example: test_shrinking(
num=0,
)
Trying example: test_shrinking(
num=-1,
)
Trying example: test_shrinking(
num=-2,
)
Trying example: test_shrinking(
num=3,
)
Falsifying example: test_shrinking(
num=-3,
)
Here, we tested the expression num >= -2 against a pool of integers. Hypothesis started with num = -4475302896957925906, a rather large number, as the first fail case. It then shrinked the values of num until Hypothesis found the value num = -3 as the smallest fail case.
Using Hypothesis with FastAPI
Hypothesis has proven to be a simple yet powerful testing tool. Let's see how we can use it with FastAPI.
# server.py
import uvicorn
from fastapi import FastAPI
app = FastAPI()
@app.get("/api/{s}")
def homepage(s: int):
return {"message": s * s}
if __name__ == "__main__":
uvicorn.run(app)
So, the /api/{s} route takes a URL parameter called s that should be an integer.
# test_server.py
from hypothesis import given, strategies as st
from fastapi.testclient import TestClient
from server import app
client = TestClient(app)
@given(st.integers())
def test_home(s):
res = client.get(f"/api/{s}")
assert res.status_code == 200
assert res.json() == {"message": s * s}
Like before we used the integers Strategy to generate random integers, positive and negative, for testing.
Schemathesis
Schemathesis is a modern API testing tool based on the OpenAPI and GraphQL specifications. It uses Hypothesis under the hood to apply property-based testing to API schemas. In other words, given a schema, Schemathesis can automatically generate test cases for you. Since FastAPI is based on OpenAPI standards, Schemathesis works well with it.
If you run the above server.py file and navigate to http://localhost:8000/openapi.json, you should see the OpenAPI specification generated by FastAPI. It defines all the endpoints along with their input types. Using this spec, Schemathesis can be used to generate test data.
Quick Start with the CLI
Install:
$ pip install schemathesis
Once installed, the simplest way to run tests is via the schemathesis command. With Uvicorn running in one terminal window, open a new window and run:
$ schemathesis run http://localhost:8000/openapi.json
You should see:
========================= Schemathesis test session starts ========================
Schema location: http://localhost:8000/openapi.json
Base URL: http://localhost:8000/
Specification version: Open API 3.0.2
Workers: 1
Collected API operations: 1
GET /api/{s} . [100%]
===================================== SUMMARY =====================================
Performed checks:
not_a_server_error 100 / 100 passed PASSED
================================ 1 passed in 0.61s ================================
Notice how this only checked for not_a_server_error. Schemathesis has five built-in checks:
not_a_server_error: response has 5xx HTTP statusstatus_code_conformance: response status is not defined in the API schemacontent_type_conformance: response content type is not defined in the API schemaresponse_schema_conformance: response content does not conform to the schema defined for this specific responseresponse_headers_conformance: response headers does not contain all defined headers.
You can perform all built-in checks with the --checks all option:
$ schemathesis run --checks all http://localhost:8000/openapi.json
========================= Schemathesis test session starts ========================
Schema location: http://localhost:8000/openapi.json
Base URL: http://localhost:8000/
Specification version: Open API 3.0.2
Workers: 1
Collected API operations: 1
GET /api/{s} . [100%]
===================================== SUMMARY =====================================
Performed checks:
not_a_server_error 100 / 100 passed PASSED
status_code_conformance 100 / 100 passed PASSED
content_type_conformance 100 / 100 passed PASSED
response_headers_conformance 100 / 100 passed PASSED
response_schema_conformance 100 / 100 passed PASSED
================================ 1 passed in 0.87s ================================
Additional Options
You can test a specific endpoint or HTTP method rather than the whole application:
$ schemathesis run --endpoint /api/. http://localhost:8000/openapi.json
$ schemathesis run --method GET http://localhost:8000/openapi.json
A max response time can be used to help flesh out edge cases that may slow down the endpoints. The time is in milliseconds.
$ schemathesis run --max-response-time=50 HTTP://localhost:8000/openapi.json
Do some of your endpoints require authorization?
$ schemathesis run -H "Authorization: Bearer TOKEN" http://localhost:8000/openapi.json
$ schemathesis run -H "Authorization: ..." -H "X-API-Key: ..." HTTP://localhost:8000/openapi.json
You can use multiple workers to speed up the tests:
$ schemathesis run --workers 8 http://localhost:8000/openapi.json
Normally, Schemathesis generates random data for each endpoint. Stateful tests make sure that the data comes from previous tests/responses:
$ schemathesis run --stateful=links http://localhost:8000/openapi.json
Finally, replaying tests is simple since each test case is associated with a seed value. When a test case fails, it'll provide the seed so that the you can reproduce the failed case:
$ schemathesis run http://localhost:8000/openapi.json
============================ Schemathesis test session starts ============================
platform Darwin -- Python 3.10.6, schemathesis-3.17.2, hypothesis-6.54.4,
hypothesis_jsonschema-0.22.0, jsonschema-4.15.0
rootdir: /hypothesis-examples
hypothesis profile 'default' ->
database=DirectoryBasedExampleDatabase('/hypothesis-examples/.hypothesis/examples')
Schema location: http://localhost:8000/openapi.json
Base URL: http://localhost:8000/
Specification version: Open API 3.0.2
Workers: 1
collected endpoints: 1
GET /api/{s} F [100%]
======================================== FAILURES ========================================
_____________________________________ GET: /api/{s} ______________________________________
1. Received a response with 5xx status code: 500
Path parameters : {'s': 0}
Run this Python code to reproduce this failure:
requests.get('http://localhost:8000/api/0', headers={'User-Agent': 'schemathesis/2.6.0'})
Or add this option to your command line parameters:
--hypothesis-seed=135947773389980684299156880789978283847
======================================== SUMMARY =========================================
Performed checks:
not_a_server_error 0 / 3 passed FAILED
=================================== 1 passed in 0.10s ====================================
Then, to reproduce, run:
$ schemathesis run http://localhost:8000/openapi.json --hypothesis-seed=135947773389980684299156880789978283847
Python Tests
You can use Schemathesis inside your tests as well:
import schemathesis
schema = schemathesis.from_uri("http://localhost:8000/openapi.json")
@schema.parametrize()
def test_api(case):
case.call_and_validate()
Schemathesis also supports making calls directly to ASGI (i.e., Uvicorn and Daphne) and WSGI (i.e., Gunicorn and uWSGI) applications instead of over the network:
import schemathesis
from schemathesis.specs.openapi.loaders import from_asgi
from server import app
schema = from_asgi("/openapi.json", app)
@schema.parametrize()
def test_api(case):
response = case.call_asgi()
case.validate_response(response)
Conclusion
Hopefully you can see just how powerful property-based testing can be. It can make your code more robust without reducing test readability from boilerplate test code. Features like shrinking to find the simplest failure cases and replaying improve productivity. Property-based testing will decrease time spent on writing manual tests while increasing test coverage.
A property-based testing tool like Hypothesis should be part of nearly every Python workflow. Schemathesis is used for automated API testing based on OpenAPI standards, which is very useful when coupled with an API-focused framework like FastAPI.