Tips and Tricks

Test interactive Python examples with doctest


Python tip:

You can use the doctest module to run interactive code examples inside your docstrings.

It notifies you If any of the examples don't return the expected value.

πŸ‘‡

def sum_ab(a, b):
    """
    Sum numbers a and b
    >>> sum_ab(1, 3)
    4
    >>> sum_ab(-41, 50)
    1
    :return: sum of 2 numbers
    """
    return a + b


# python -m doctest example.py
# **********************************************************************
# File "example.py", line 4, in example.sum_ab
# Failed example:
#     sum_ab(-41, 50)
# Expected:
#     1
#     :return: sum of 2 numbers
# Got:
#     9
# **********************************************************************
# 1 items had failures:
#    1 of   2 in example.sum_ab
# ***Test Failed*** 1 failures.

Python itertools.zip_longest() Example


Python tip:

You can use zip_longest from itertools to zip multiple iterables of different lengths.

"Missing" values from shorter iterables are replaced by fillvalue.

An example πŸ‘‡

from itertools import zip_longest

students = ["Bob", "Ann", "John", "Marry", "Daisy", "Amy"]
grades = ["A", "A+", "D"]

for student, grade in zip_longest(students, grades, fillvalue="-"):
    print(student, grade)

How do I use itertools.groupby()?


Python tip:

You can use groupby from itertools to group elements.

  1. Returns - an iterator of consecutive keys and groups from the iterable.
  2. Keys - values returned from key function
  3. Groups - iterator returns values for group

For example, to group tax IDs πŸ‘‡

import re
from itertools import groupby

tax_ids = [
    "DE123456789",
    "ATU99999999",
    "BG999999999",
    "IT12345678900",
    "DE987654321",
]

grouped_tax_ids = groupby(
    tax_ids, key=lambda tax_id: re.match(r"[A-Z]{2}", tax_id).group()
)

for country, country_tax_ids in grouped_tax_ids:
    print(country, list(country_tax_ids))

# DE ['DE123456789']
# AT ['ATU99999999']
# BG ['BG999999999']
# IT ['IT12345678900']
# DE ['DE987654321']

Python's Itertools.cycle()


Python tip:

Do you need to repeat a sequence over and over again?

Use cycle from itertools πŸ‘‡

from itertools import cycle

chord_sequence = cycle(["G", "D", "e", "C"])

song_chords = [next(chord_sequence) for _ in range(16)]

print(song_chords)
# ['G', 'D', 'e', 'C', 'G', 'D', 'e', 'C', 'G', 'D', 'e', 'C', 'G', 'D', 'e', 'C']

Django - Use foreign key values directly


Django tip:

Use foreign key values directly.

Make sure to use the foreign key property instead of the foreign object ID.

  • DON'T: Song.objects.get(id=1).singer.id
  • DO: Song.objects.get(id=1).singer_id
from django.db import connection, reset_queries

from .models import Song

reset_queries()

# makes an extra SQL query to load the singer object:
Song.objects.get(id=1).singer.id
len(connection.queries) # => 2

reset_queries()

# an extra SQL query is not required:
Song.objects.get(id=1).singer_id
len(connection.queries) # => 1

Python - itertools.count()


Python tip:

You can use count from itertools to make an iterator that returns evenly spaced values:

itertools.count(start=0, step=1)

For example, to generate all odd numbers greater or equal to 101πŸ‘‡

from itertools import count

odd_numbers = count(101, 2)

# first ten elements
for _ in range(10):
    print(next(odd_numbers))

"""
101
103
105
107
109
111
113
115
117
119
"""

Python - itertools.combinations()


Python tip:

You can generate all combinations of elements from an iterable of length n by using combinations from itertools.

For example, you can generate all possible match pairs πŸ‘‡

from itertools import combinations

players = ["John", "Daisy", "Marry", "Bob"]

print(list(combinations(players, 2)))

"""
[
    ('John', 'Daisy'),
    ('John', 'Marry'),
    ('John', 'Bob'),
    ('Daisy', 'Marry'),
    ('Daisy', 'Bob'),
    ('Marry', 'Bob'),
]
"""

Function overloading with singledispatchmethod from functools


Python tip (>=3.8):

You can use singledispatchmethod to implement different (re)actions based on the first (non-self, non-cls) function's argument type.

For each type, you register an overloaded implementation.

For example, you can handle different events πŸ‘‡

from functools import singledispatchmethod


class Event:
    pass


class UserSignedUp(Event):
    def __init__(self, email):
        self.email = email


class CustomerCanceledSubscription(Event):
    def __init__(self, email):
        self.email = email


class EventHandler:
    @singledispatchmethod
    def handle_event(event: Event):
        pass  # do nothing

    @handle_event.register
    def _(self, event: UserSignedUp):
        print(f"I will prepare resources for newly signed up user: {event.email}")

    @handle_event.register
    def _(self, event: CustomerCanceledSubscription):
        print(f"I will disable resources for customer:  {event.email}")


events = [
    UserSignedUp(email="[email protected]"),
    CustomerCanceledSubscription(email="[email protected]"),
]

handler = EventHandler()
for event in events:
    handler.handle_event(event)

# I will prepare resources for newly signed up user: [email protected]
# I will disable resources for customer:  [email protected]

Python functools - total_ordering()


Python tip:

You can create an orderable class (assuming the type is totally ordered) with __eq__, one other comparison methods (__ne__, __lt__, __le__, __gt__, __ge__), and the total_ordering decorator.

An exampleπŸ‘‡

from functools import total_ordering


@total_ordering
class Sample:
    def __init__(self, value):
        self.value = value

    def __lt__(self, other):
        return self.value < other.value


x = Sample(3)
y = Sample(4)

for expression in ["x == y", "x != y", "x < y", "x <= y", "x > y", "x >= y"]:
    result = eval(expression)
    print(f"{expression} is {result}")


"""
x == y is False
x != y is True
x < y is True
x <= y is True
x > y is False
x >= y is False
"""

Without it, you'd need to define all of the comparison methods.

Python functools - cached_property()


Python tip (>=3.8):

You can use cached_property from functools to cache the results of a class attribute for the life of the instance. It's useful if you have a property that is expensive to compute and doesn't need to change.

An example πŸ‘‡

import io
import pathlib
from functools import cached_property

from pikepdf import Pdf


class PDF:
    def __init__(self, file_bytes):
        self.file_bytes = file_bytes

    @cached_property
    def number_of_pages(self):
        return len(Pdf.open(io.BytesIO(self.file_bytes)).pages)


pdf = PDF(pathlib.Path("file.pdf").read_bytes())
print(pdf.number_of_pages)

What's the difference between select_related and prefetch_related in Django?


Django tip:

Select_related vs prefetch_related

  • Use select_related() on OneToOneField or ForeignKey when you need a single object
  • Use prefetch_related() on ManyToManyFields or reverse relations when you need many objects

πŸ‘‡

class Singer(models.Model):
    first_name = models.CharField(max_length=40)
    last_name = models.CharField(max_length=40)


class Song(models.Model):
    name = models.CharField(max_length=100)
    singer = models.ForeignKey(Singer, related_name="songs")


Song.objects.select_related("singer").all()  # Forward ForeignKey relationship
Singer.objects.select_related("song").all()  # Backward ForeignKey relationship

Provision and manage remote Docker hosts with Docker Machine


Docker tip:

Use Docker Machine to quickly provision a remote machine for testing

https://docs.docker.com/machine/

# spin up an ec2 instance with docker engine installed
$ docker-machine create \
    --driver amazonec2 \
    --amazonec2-open-port 8000 \
    --amazonec2-region us-east-1 \
    --amazonec2-instance-type "t2.micro" \
    sandbox

# point your docker daemon at the remote docker engine
$ docker-machine env sandbox
$ eval $(docker-machine env sandbox)

# build the images and spin up the containers on the remote machine
$ docker-compose up -d --build

# ssh certs are auto generated, so you can easily ssh to the remote
$ docker-machine ssh sandbox

# you can also copy files to/ from the machine
$ docker-machine scp docker-machine scp sandbox:~/foo.txt ~/
$ docker-machine scp docker-machine scp foo.tar.gz sandbox:/home/ubuntu

Python - function overloading with singledispatch


Python tip:

You can use singledispatch to implement different (re)actions based on the first function's argument type

For each type, you register an overloaded implementation

For example, you can handle different events like so:

from functools import singledispatch


class Event:
    pass


class UserSignedUp(Event):
    def __init__(self, email):
        self.email = email


class CustomerCanceledSubscription(Event):
    def __init__(self, email):
        self.email = email


@singledispatch
def handle_event(event: Event):
    pass  # do nothing


@handle_event.register(UserSignedUp)
def _(event: UserSignedUp):
    print(f"I will prepare resources for newly signed up user: {event.email}")


@handle_event.register(CustomerCanceledSubscription)
def _(event: CustomerCanceledSubscription):
    print(f"I will disable resources for customer:  {event.email}")


events = [
    UserSignedUp(email="[email protected]"),
    CustomerCanceledSubscription(email="[email protected]"),
]

for event in events:
    handle_event(event)

# I will prepare resources for newly signed up user: [email protected]
# I will disable resources for customer:  [email protected]

Caching in Python with lru_cache


Python tip:

Improve the performance of a function with the lru_cache decorator from functools:

@functools.lru_cache(maxsize=128, typed=False)
  • maxsize: max number of stored arguments
  • typed: different types will be cached separately (i.e., 3 != 3.0)
  • function args and kwargs must be hashable
  • perfect for when you need to periodically call an expensive function with the same arguments

Example:

import timeit
from functools import lru_cache

import requests


@lru_cache(maxsize=10, typed=False)
def open_web_page(website_url):
    requests.get(website_url)


without_cache = """
import requests

def open_web_page(website_url):
    requests.get(website_url)

urls = [
    'https://testdriven.io',
    'https://testdriven.io',
    'https://google.com',
    'https://google.com',
]

for url in urls:
    open_web_page(url)
"""

with_cache = """
from functools import lru_cache

import requests

@lru_cache(maxsize=10, typed=False)
def open_web_page(website_url):
    requests.get(website_url)

urls = [
    'https://testdriven.io',
    'https://testdriven.io',
    'https://google.com',
    'https://google.com',
]
for url in urls:
    open_web_page(url)
"""


print(timeit.timeit(without_cache, number=5))
# => 7.195018381

print(timeit.timeit(with_cache, number=5))
# => 3.6599477370000004

Use the Django auth system with a Single Page App (SPA)


Django tip:

Coupling Django with a front-end framework like React, Vue, or Angular? Use session cookies for auth (even cross-domain).

Why?

  1. It's easier since Django has a powerful built-in auth system
  2. It's safer than using JWTs and localStorage

https://testdriven.io/blog/django-spa-auth/

Partial functions in Python with partial from functools


Python tip:

You can use partial from functools to freeze a certain number of arguments from a function and create a new, simplified function.

It returns a new partial object which when called will behave like a function called with the positional args and keywords.

An example πŸ‘‡

from functools import partial

all_users = [
    {"email": "[email protected]"},
    {"email": "[email protected]"},
    {"email": "[email protected]"},
]


def _sort_users_by_email(users, sort_order):
    asc_dsc_to_reverse = {
        "ascending": True,
        "descending": False,
    }

    return sorted(
        users, key=lambda user: user["email"], reverse=asc_dsc_to_reverse[sort_order]
    )


_sort_users_by_email_ascending = partial(_sort_users_by_email, sort_order="ascending")
_sort_users_by_email_descending = partial(_sort_users_by_email, sort_order="descending")


print(_sort_users_by_email_ascending(all_users))
# [{'email': '[email protected]'}, {'email': '[email protected]'}, {'email': '[email protected]'}]

print(_sort_users_by_email_descending(all_users))
# [{'email': '[email protected]'}, {'email': '[email protected]'}, {'email': '[email protected]'}]

Get the unique values from a list of dictionaries


Python tip:

You can use a dictionary comprehension to create a list of dictionaries unique by value on a selected key

users = [
    {"name": "John Doe", "email": "[email protected]"},
    {"name": "Mary Doe", "email": "[email protected]"},
    {"name": "Mary A. Doe", "email": "[email protected]"},
]


print(list({user["email"]: user for user in users}.values()))

"""
[
    {'name': 'John Doe', 'email': '[email protected]'},
    {'name': 'Mary A. Doe', 'email': '[email protected]'}
]

The same key can occur only once inside a dictionary.
Every element from a list is assigned to a key with a value of email.
Each new element with the same email overwrites any existing ones.
Therefore, only one element is left with the same value on the key.

.values() returns a dict_values object containing values from the dictionary,
list() converts the dict_values object back to a list.
"""

Subtracting two lists in Python


Python tip:

To subtract two lists (element by element), you can use a list comprehension and zip.

zip iterates over both lists -> you receive elements with the same index.

The list comprehension then creates a new list containing subtracted values.

πŸ‘‡

first = [1, 5, 7]
second = [5, 3, 0]

print([xi - yi for xi, yi in zip(first, second)])
# => [-4, 2, 7]

Handling missing keys with Python's defaultdict


Python tip:

You can use defaultdict to provide a default value for missing keys when accessing or assigning:

defaultdict(default_factory)

default_factory is a function used to initialize the value for the missing key (None is set by default).

If you pass the list constructor as the default_factory, a defaultdict is created with the values that are list.

For example:

import json
from collections import defaultdict

products = [
    {"name": "Cold beer", "price": 3.5, "category": "Beer"},
    {"name": "Coffee", "price": 1.5, "category": "Warm drinks"},
    {"name": "Cappuccino", "price": 2.0, "category": "Warm drinks"},
    {"name": "Chicken Sandwich", "price": 3.5, "category": "Snacks"},
    {"name": "Vegan Sandwich", "price": 3.5, "category": "Snacks"},
]

products_by_group = defaultdict(list)

for product in products:
    products_by_group[product["category"]].append(product)

print(json.dumps(products_by_group, indent=2))

"""
{
  "Beer": [
    {
      "name": "Cold beer",
      "price": 3.5,
      "category": "Beer"
    }
  ],
  "Warm drinks": [
    {
      "name": "Coffee",
      "price": 1.5,
      "category": "Warm drinks"
    },
    {
      "name": "Cappuccino",
      "price": 2.0,
      "category": "Warm drinks"
    }
  ],
  "Snacks": [
    {
      "name": "Chicken Sandwich",
      "price": 3.5,
      "category": "Snacks"
    },
    {
      "name": "Vegan Sandwich",
      "price": 3.5,
      "category": "Snacks"
    }
  ]
}
"""

Python Counting - finding the most common occurence


Python tip:

You can use Counter from the collections module to find the most common element in a:

  • string - character with the most occurrences
  • dict - key with highest value
  • list - element with the most occurrences

most_common(n) -> returns n most common element

from collections import Counter

counter = Counter("TestDriven.io tutorials")
print(counter.most_common(1))  # character with the most occurrences
# => [('t', 3)]

counter = Counter({"red": 2, "green": 10, "blue": 7})
print(counter.most_common(1))  # key with the highest value
# => [('green', 10)]

counter = Counter([0, 2, 1, 1, 4, 10, 33, 21, 12, 10, 1])
print(counter.most_common(1))  # element with the most occurrences
# => [(1, 3)]

Python type hints - typing.Literal


Python tip:

You can use Literal to indicate that value can be one of the provided literals.

Static type checkers will report an error when the value doesn't match one of the provided literals.

https://mypy.readthedocs.io/en/stable/literal_types.html#literal-types

from typing import Literal

STATUS = Literal["ACTIVE", "DISABLED"]


class User:
    def __init__(self, username: str, status: STATUS):
        self.username = username
        self.status = status


user = User("[email protected]", "CREATED")

"""
mypy example.py
example.py:12: error: Argument 2 to "User" has incompatible type "Literal['CREATED']";
expected "Union[Literal['ACTIVE'], Literal['DISABLED']]"
Found 1 error in 1 file (checked 1 source file)
"""

Python type hints - creating a type alias


Python tip:

You can create a type alias to make your code more readable.

from typing import List

Vector = List[float]


def scale(scalar: float, vector: Vector) -> Vector:
    return [scalar * num for num in vector]

Python Type Hints - typing.Union


Python tip:

Use Union from the typing module to allow any of the listed types.

from typing import Union


def sum_ab(a: Union[int, float], b: Union[int, float]) -> Union[int, float]:
    return a + b

Python Type Hints - typing.TypedDict


Python (>=3.8) tip:

You can subclass TypedDict to create a type for dictionaries with fixed keys.

Static type checking will report an error when there are extra or missing keys.

from typing import TypedDict


class Song(TypedDict):
    name: str
    year: int


song: Song = {"name": "Enter Sandman", "year": 1991, "band": "Metallica"}

"""
mypy song.py
song.py:9: error: Extra key "band" for TypedDict "Song"
Found 1 error in 1 file (checked 1 source file)
"""

Python's final qualifier


Python (>=3.8) tip:

You can use the final decorator to declare that a:

  1. Method should not be overridden
  2. Class should not be subclassed

You can also use the Final annotation to declare that a variable or attribute should not be reassigned, redefined, or overridden. It can be used as indicator to another developer that the given variable should be treated as a constant:

from typing import Final 

_PI: Final =  3.14

A static type check will report an error when it's violated.

https://mypy.readthedocs.io/en/stable/final_attrs.html

from typing import Final, final

# example 1


@final
class Cat:
    @final
    def meow(self):
        return "Meow"


class WildCat(Cat):
    def meow(self):
        return "Grrr"


# mypy cat.py
# error: Cannot inherit from final class "Cat"


# example 2


class Foo:
    @final
    def bar(self):
        return "baz"


def foobar():
    return "foobar"


new = Foo()
new.bar = foobar

# mypy foo.py
# error: Cannot assign to final attribute "bar"


# example 3

favorite_color: Final[str] = "red"
favorite_color = "blue"

# mypy foo.py
# error: Cannot assign to final name "favorite_color"

Python's platform module


Python tip:

You can check the type of machine and platform that your program is running from with platform.machine() and platform.platform(), respectively.

An exampleπŸ‘‡

import platform

print(platform.machine())
# => x86_64

print(platform.platform())
# => macOS-10.15.6-x86_64-i386-64bit

Problem solving - what have you tried?


When stuck, spend at least 30 minutes but no more than 60 trying to solve the problem before asking for help. Also, when you ask for help make sure you can answer the question, "What have you tried?”.