The Definitive Guide to Celery and Django
Learn how to add Celery to a Django application to provide asynchronous task processing.
Learning Objectives
Here are the key concepts that you will learn in each part of this course.
Part 1
- Explain why you may want to use a task queue like Celery
- Describe the basic producer/consumer model and how it relates to Celery
- Set up Celery with Django
- Use Docker Compose to create and manage Django, Postgres, Redis, and Celery
- Implement auto-reload problem
- Debug a Celery task with rdb
- Process Django form submissions with a Celery worker
- Handle complicated logic triggered by a webhook notification with a Celery worker
- Retry a failed Celery task with the
retry
method - Use Django Channels to add WebSocket support to a Django application
- Schedule a Django management command to run periodically with Celery Beat
- Describe what a database transaction is and how to use it in Django
- Explain why you might get a
DoesNotExist
error in a Celery worker and how to solve it - Prevent a task from executing before the database commits a transaction
Part 2
- Explain how the Celery logger works in conjunction with Django
- Set up the Celery logger inside a Celery Task
- Customize the behavior of Celery's Logger
- Send Celery logs to Papertrail
- Test Celery tasks in Celery's eager mode
- Test a Django app with pytest and pytest-django
- Use pytest fixtures and test markers
- Generate test data with factory_boy and pytest-factoryboy
- Practice Test-Driven Development with pytest as you test a Celery task
- Create and use a custom Celery task decorator
- Use pytest to test a custom task decorator
- Deploy Django and Celery to DigitalOcean
- Monitor a Celery app with Flower
- Configure Container Advisor and Prometheus for monitoring
- Use Kombu to produce and consume messages through a message broker
Tools and Technologies
Core
- Python
- Django
- Celery
- Django Channels
- Flower
- Docker
- Postgres
- Redis
- RabbitMQ
- Gunicorn
- Uvicorn
Testing and Monitoring
- pytest
- factory_boy
- Coverage.py
- Container Advisor
- Prometheus
Services
- Papertrail
- DigitalOcean
This course uses Celery v4.4.7 since Flower does not support Celery 5.
Prerequisites
This is not a beginner course. It's designed for the advanced-beginner -- someone with at least six months of web development experience. Before beginning, you should have some familiarity with the following topics. Refer to these resources for more info:
Topic(s) | Resource(s) |
---|---|
Docker and Docker Compose | Test-Driven Development with Python, Django, and Docker, Get started with Docker, and Get started with Docker Compose |
Django | Test-Driven Development with Python, Django, and Docker |
Looking for a solid introduction to Django? Check out Django for Beginners. Highly recommended!
Info
- Current version: 1.0.0
- Last updated: January 4th, 2021
- Author: Michael Yin
What will you learn?
Part 1
In this first part, you'll learn how to configure Celery to work with Django and Docker in order to process tasks in the background asynchronously. We'll then look at how to retry failed tasks, debug tasks, and ensure tasks work with Django database transactions correctly. Finally, we'll add WebSocket support to Django with Django Channels.
Part 2
In part 2, we'll dive into testing with the standard Django testing framework and pytest. We'll also look at a Test-Driven Development (TDD) workflow. You'll then deploy the application to DigitalOcean and learn how to properly monitor Celery tasks with Flower, Container Advisor, and Prometheus. Finally, we'll look at some Celery best practices and tips for performance tuning.
Table of Contents
Table of contents from The Definitive Guide to Celery and Django:
Part 1
- Introduction
- Changelog
- Getting Started
- Dockerizing Celery and Django
- Setting Up Auto-reload
- Debugging a Celery Task
- Third-party Services
- Django Channels and Celery
- Periodic Tasks
- Multiple Queues and Task Routing
- Retrying Failed Tasks
- Database Transactions
Part 2
- Logging
- Django Test Framework
- Pytest
- Test-Driven Development
- Task Decorator
- Deployment
- Monitoring
- Config and Best Practices
- Performance Tuning
- Bonus: Kombu
- Conclusion
Get the full course:
Join our mailing list to be notified about course updates and new tutorials.