The Definitive Guide to Celery and Django


Learn how to add Celery to a Django application to provide asynchronous task processing.

Learning Objectives

Here are the key concepts that you will learn in each part of this course.

Part 1

  1. Explain why you may want to use a task queue like Celery
  2. Describe the basic producer/consumer model and how it relates to Celery
  3. Set up Celery with Django
  4. Use Docker Compose to create and manage Django, Postgres, Redis, and Celery
  5. Implement auto-reload problem
  6. Debug a Celery task with rdb
  7. Process Django form submissions with a Celery worker
  8. Handle complicated logic triggered by a webhook notification with a Celery worker
  9. Retry a failed Celery task with the retry method
  10. Use Django Channels to add WebSocket support to a Django application
  11. Schedule a Django management command to run periodically with Celery Beat
  12. Describe what a database transaction is and how to use it in Django
  13. Explain why you might get a DoesNotExist error in a Celery worker and how to solve it
  14. Prevent a task from executing before the database commits a transaction

Part 2

  1. Explain how the Celery logger works in conjunction with Django
  2. Set up the Celery logger inside a Celery Task
  3. Customize the behavior of Celery's Logger
  4. Send Celery logs to Papertrail
  5. Test Celery tasks in Celery's eager mode
  6. Test a Django app with pytest and pytest-django
  7. Use pytest fixtures and test markers
  8. Generate test data with factory_boy and pytest-factoryboy
  9. Practice Test-Driven Development with pytest as you test a Celery task
  10. Create and use a custom Celery task decorator
  11. Use pytest to test a custom task decorator
  12. Deploy Django and Celery to DigitalOcean
  13. Monitor a Celery app with Flower
  14. Configure Container Advisor and Prometheus for monitoring
  15. Use Kombu to produce and consume messages through a message broker

Tools and Technologies

Core

  1. Python
  2. Django
  3. Celery
  4. Django Channels
  5. Flower
  6. Docker
  7. Postgres
  8. Redis
  9. RabbitMQ
  10. Gunicorn
  11. Uvicorn

Testing and Monitoring

  1. pytest
  2. factory_boy
  3. Coverage.py
  4. Container Advisor
  5. Prometheus

Services

  1. Papertrail
  2. DigitalOcean

This course uses Celery v4.4.7 since Flower does not support Celery 5.

Prerequisites

This is not a beginner course. It's designed for the advanced-beginner -- someone with at least six months of web development experience. Before beginning, you should have some familiarity with the following topics. Refer to these resources for more info:

Topic(s) Resource(s)
Docker and Docker Compose Test-Driven Development with Python, Django, and Docker, Get started with Docker, and Get started with Docker Compose
Django Test-Driven Development with Python, Django, and Docker

Looking for a solid introduction to Django? Check out Django for Beginners. Highly recommended!

Info

  • Current version: 1.0.0
  • Last updated: January 4th, 2021
  • Author:  Michael Yin



What will you learn?

Part 1

In this first part, you'll learn how to configure Celery to work with Django and Docker in order to process tasks in the background asynchronously. We'll then look at how to retry failed tasks, debug tasks, and ensure tasks work with Django database transactions correctly. Finally, we'll add WebSocket support to Django with Django Channels.

Part 2

In part 2, we'll dive into testing with the standard Django testing framework and pytest. We'll also look at a Test-Driven Development (TDD) workflow. You'll then deploy the application to DigitalOcean and learn how to properly monitor Celery tasks with Flower, Container Advisor, and Prometheus. Finally, we'll look at some Celery best practices and tips for performance tuning.



Table of Contents

Table of contents from The Definitive Guide to Celery and Django:


Part 1
Part 2
  • Logging
  • Django Test Framework
  • Pytest
  • Test-Driven Development
  • Task Decorator
  • Deployment
  • Monitoring
  • Config and Best Practices
  • Performance Tuning
  • Bonus: Kombu
  • Conclusion

Get the full course:




Join our mailing list to be notified about course updates and new tutorials.