Celery Integration in Django

In this blog, we’ll cover following concepts-

1. Asynchronous execution of tasks and task scheduling
2. How Celery works?
3. Message brokers and workers
4. Step-by-step process for celery integration in django

1. Asynchronous execution of tasks and task scheduling

Asynchronous execution lets us to move to another task without finishing the first task at hand whereas in a synchronous fashion we need to complete the running task before starting another one. This asynchronous execution is needed at times when we need to do heavy computation task in server side and we want to show quick response to users instead of letting them wait and stare at the screen until the process completes. It is used in cases where we need to send emails to multiple clients, performing any long running tasks and so on. Task scheduling means specifying a particular time slot for a particular task. Every task tries to finish off their work in the allotted time slot and waits for another time slot if task remains incomplete.

2. How Celery works?

Celery provides workers, a worker is a function that executes out tasks, a task if a function that is defined in our django project. Django passes these tasks to the workers and usually brokers are used to do this.

3. Message brokers and workers

Broker is a task queue that stores our tasks. After broker gets the tasks from django, it puts the tasks in a queue and then it starts to pass these tasks to workers. After worker completes its tasks, it puts the results in results backend and then we can fetch those results from django app. Usually the same broker is used to store the results.

Message brokers examples => Redis, RabbitMQ

4. Step-by-step process for celery integration in django

i. Install the requirements

1
pip install celery, eventlet, redis, django, djangorestframework

ii. Setup the django project

1
django-admin startproject emailModule

1
cd emailModule

1
python manage.py startapp emailApp

-append ‘restframework’ and ‘emailApp’ in InstalledApps list in settings.py file

iii. create emailModule/emailModule/celery.py and include the following

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from celery import Celery

import os

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'emailModule.settings')

app = Celery('emailModule')

app.config_from_object('django.conf:settings', namespace = 'CELERY')

app.autodiscover_tasks()

@app.task(bind = True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

iv. in emailModule/emailModule/init.py, add below code

1
2
3
from .celery import app as celery_app

__all__ = ['celery_app']

v. in emailModule/emailModule/setting.py, add below code

1
2
3
4
5
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['json']
# CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'

vi. add email settings in settings.py, add your email and password

1
2
3
4
5
EMAIL_HOST='smtp.gmail.com'
EMAIL_HOST_USER='<your_email>'
EMAIL_HOST_PASSWORD='<your_password>'
EMAIL_PORT=587
EMAIL_USE_TLS=True

-also don’t forget to turn off 2-step verification for this email and also turn on less secure apps in gmail settings to make this email work properly

vii. create emailApp/task.py to include all the asynchronous tasks

1
2
3
4
5
6
7
8
9
10
11
12
from celery import shared_task
from django.core.mail import send_mail
from django.conf import settings

@shared_task
def send_email_to_all(receiver_list):
    subject = "Test Message"
    message = "Hello World"
    send_mail(subject=subject, message = message, from_email = settings.EMAIL_HOST_USER,
     recipient_list = receiver_list, fail_silently = False)
    return None

viii. use this asynchronous task scheduler ; in emailApp/views.py, add below code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from rest_framework.views import APIView
from rest_framework.response import Response

from emailApp.tasks import send_email_to_all

# Create your views here.

class SendMailView(APIView):

    def post(self, request, *args, **kwargs):
        '''
        For post email
        '''
        email_list = request.data.get('email_list')
        send_email_to_all.delay(email_list)
        return Response({'message': 'Ok'})

ix. add urls in emailModule/emailModule/urls.py

1
2
3
4
5
6
from emailApp.views import SendMailView

urlpatterns = [
    ...,
    path('api/send_mail/', SendMailView.as_view()),
]

x. Run celery, redis-server and django server to make an api call to send email

-open a new terminal and run below command

1
celery -A emailModule worker -l info -P eventlet

-open a new terminal to run redis

1
sudo systemctl start redis-server

-open a new terminal to run django server

1
python manage.py runserver

Now we can call that api with appropriate email list to send email asynchronously in django.

If you want to refer to the code, here is the link to the project.

Thank you.