Jan Giacomelli
Python 3.11 was released on October 24th, 2021. This article looks at the most interesting new additions to the Python language that will help to make your Python code even cleaner:
- Faster CPython
- Improved type hints
- Better error messages
- Exception notes
- TOML library
Contents
Installing Python 3.11
If you have Docker, you can quickly spin up a Python 3.11 shell to play around with the examples in this article like so:
$ docker run -it --rm python:3.11
Not using Docker? We recommend installing Python 3.11 with pyenv:
$ pyenv install 3.11.0
You can learn more about managing Python with pyenv from the Modern Python Environments - dependency and workspace management article.
Faster CPython
Python 3.11 is faster than ever! As stated in the release notes, Python 3.11 is 10 - 60% faster than Python 3.10. On average, it's 25% faster. Python's core developers did a great job improving execution time in a number of areas.
Faster Startup
In terms of startup, rather than Reading __pycache__ -> Unmarshall -> Heap allocated code object -> Evaluate, Python 3.11 uses a new process. Core modules essential for the startup of the Python interpreter are now frozen in the interpreter -- their code is statically allocated. The new flow is Statically allocated code object -> Evaluate. The latter is 10 - 15% faster.
Faster Runtime
Every time you call a function in Python, a frame gets created. It holds information about function execution. The core developers simplified their creation process, internal information, and memory allocation. Another improvement occurred with inline function calls. From now on, most of the Python function calls consume no C stack space.
Maybe the most important improvement in this regard is PEP 659: Specializing Adaptive Interpreter. This PEP is one of the key elements for fast Python. Its main idea is that Python code has regions where types rarely change. Within those regions, Python can optimize operations by using more specialized types when it sees that some operation is always done only for certain types of data. For example, instead of using general multiplication, it can use multiplication for integers if only integers are being used. Another example is the direct call of underlying C implementations for common built-in functions like len and str instead of going through the internal calling convention.
You can observe this by checking generated bytecode:
import dis
from random import random
def dollars_to_pounds(dollars):
return 0.87 * dollars
dis.dis(dollars_to_pounds, adaptive=True)
# 5 0 RESUME 0
#
# 6 2 LOAD_CONST 1 (0.87)
# 4 LOAD_FAST 0 (dollars)
# 6 BINARY_OP 5 (*)
# 10 RETURN_VALUE
for _ in range(8):
dollars_to_pounds(random() * 100)
dis.dis(dollars_to_pounds, adaptive=True)
# 5 0 RESUME_QUICK 0
#
# 6 2 LOAD_CONST__LOAD_FAST 1 (0.87)
# 4 LOAD_FAST 0 (dollars)
# 6 BINARY_OP_MULTIPLY_FLOAT 5 (*) # <-- CHANGED!
# 10 RETURN_VALUE
As you can see, after 8 calls of dollars_to_pounds with a float, the bytcode is optimized. Instead of using BINARY_OP, it's using specialized BINARY_OP_MULTIPLY_FLOAT, which is faster when multiplying floats. If you'd run the same thing with an older Python version there wouldn't be any difference.
For more, refer to the official docs along with the associated PEP 659 - Specializing Adaptive Interpreter.
Improved Type Hints
As in the last couple of releases, there are improvements on the type hinting front.
Self Type
Python finally has support for a Self type. So, now you can easily type class methods and dunder methods:
from typing import Self
from dataclasses import dataclass
@dataclass
class Car:
manufacture: str
model: str
@classmethod
def from_dict(cls, car_data: dict[str, str]) -> Self:
return cls(manufacture=car_data["manufacture"], model=car_data["model"])
print(Car.from_dict({"manufacture": "Alfa Romeo", "model": "Stelvio"}))
# Car(manufacture='Alfa Romeo', model='Stelvio')
More info:
TypedDict NotRequired
Another improvement is the NotRequired type for typed dictionaries:
from typing import TypedDict, NotRequired
class Car(TypedDict):
manufacture: str
model: NotRequired[str]
car1: Car = {"manufacture": "Alfa Romeo", "model": "Stelvio"} # OK
car2: Car = {"manufacture": "Alfa Romeo"} # model (model is not required)
car3: Car = {"model": "Stelvio"} # ERROR (missing required field manufacture)
More info:
Literal String Type
There's also a new LiteralString type that allows literal strings and strings created from other literal strings. This can be used for type checking when executing SQL and shell commands to add another layer of safety for preventing injection attacks.
def run_query(sql: LiteralString) -> ...
...
def caller(
arbitrary_string: str,
query_string: LiteralString,
table_name: LiteralString,
) -> None:
run_query("SELECT * FROM students") # ok
run_query(query_string) # ok
run_query("SELECT * FROM " + table_name) # ok
run_query(arbitrary_string) # type checker error
run_query( # type checker error
f"SELECT * FROM students WHERE name = {arbitrary_string}"
)
More info:
--
There are also some other improvements related to type hints. You can read about them in the official release notes.
Better Error Messages
One of the exciting new features is more descriptive tracebacks. Python 3.10 saw some improvements made by introducing better and more descriptive errors. Python 3.11 takes it a step further with improved descriptions of the exact error position. Let's take a look at an example.
Code with syntax error:
def average_grade(grades):
return sum(grades) / len(grades)
average_grade([])
Error in Python 3.10:
return sum(grades) / len(grades)
ZeroDivisionError: division by zero
Error in Python 3.11:
return sum(grades) / len(grades)
~~~~~~~~~~~~^~~~~~~~~~~~~
ZeroDivisionError: division by zero
How about something a bit more complex?
Python 3.10:
def send_email_to_contact(contact, subject, content):
print(f"Sending email to {contact['emails']['address']} with subject={subject.title()} and {content=}")
contact = {
"first_name": "Lightning",
"last_name": "McQueen",
"emails": [
{
"address": "[email protected]",
"display_name": "Lightning McQueen"
}
]
}
send_email_to_contact(contact, "Hello", "Hello there! Long time no see.")
# print(f"Sending email to {contact['emails']['address']} with subject={subject.title()} and {content=}")
# TypeError: list indices must be integers or slices, not str
send_email_to_contact({}, "Hello", "Hello there! Long time no see.")
# print(f"Sending email to {contact['emails']['address']} with subject={subject.title()} and {content=}")
# KeyError: 'emails'
send_email_to_contact({"emails": {"address": "[email protected]"}}, None, "Hello there! Long time no see.")
# print(f"Sending email to {contact['emails']['address']} with subject={subject.title()} and {content=}")
# AttributeError: 'NoneType' object has no attribute 'title'
Python 3.11:
def send_email_to_contact(contact, subject, content):
print(f"Sending email to {contact['emails']['address']} with subject={subject.title()} and {content=}")
contact = {
"first_name": "Lightning",
"last_name": "McQueen",
"emails": [
{
"address": "[email protected]",
"display_name": "Lightning McQueen"
}
]
}
send_email_to_contact(contact, "Hello", "Hello there! Long time no see.")
# print(f"Sending email to {contact['emails']['address']} with {subject=} and {content=}")
# ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
# TypeError: list indices must be integers or slices, not str
send_email_to_contact({}, "Hello", "Hello there! Long time no see.")
# print(f"Sending email to {contact['emails']['address']} with subject={subject.title()} and {content=}")
# ~~~~~~~^^^^^^^^^^
# KeyError: 'emails'
send_email_to_contact({"emails": {"address": "[email protected]"}}, None, "Hello there! Long time no see.")
# print(f"Sending email to {contact['emails']['address']} with subject={subject.title()} and {content=}")
# ^^^^^^^^^^^^^
# AttributeError: 'NoneType' object has no attribute 'title'
As you can see, it's much easier to find out where the error is from the new traceback. The exact problematic spot in the line with exception is well-marked. In older versions, you can see only the exception itself and the line where it was raised.
More info:
Exception Notes
add_note was added to BaseExceptions. This allows you to add additional context to an exception after it was created. For example:
try:
raise ValueError()
except ValueError as exc:
exc.add_note("When this happened my dog was barking and my kids were sleeping.")
raise
# raise ValueError()
# ValueError
# When this happened my dog was barking and my kids were sleeping.
More info:
TOML Library
Python now has a library for parsing TOML files called tomllib. Its usage is very similar to that of the built-in json library.
For example, say you have the following pyproject.toml file:
[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = []
[tool.poetry.dependencies]
python = "^3.11"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
You can load the file like so:
import pprint
import tomllib
with open("pyproject.toml", "rb") as f:
data = tomllib.load(f)
pp = pprint.PrettyPrinter(depth=4)
pp.pprint(data)
"""
{'build-system': {'build-backend': 'poetry.core.masonry.api',
'requires': ['poetry-core>=1.0.0']},
'tool': {'poetry': {'authors': [],
'dependencies': {'python': '^3.11'},
'description': '',
'dev-dependencies': {},
'name': 'example',
'version': '0.1.0'}}}
"""
More info:
Conclusion
As you can see, Python 3.11 brought many exciting improvements and features. On the less exciting side, some legacy modules were deprecated and others removed altogether. Since this article covered only the most interesting new features and changes, be sure to check out the official release notes for all the changes.
Happy Pythoning!